May 26 16:00:47.297243 ip-10-0-131-126 systemd[1]: Starting Kubernetes Kubelet... May 26 16:00:47.828912 ip-10-0-131-126 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 26 16:00:47.828912 ip-10-0-131-126 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. May 26 16:00:47.828912 ip-10-0-131-126 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 26 16:00:47.828912 ip-10-0-131-126 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. May 26 16:00:47.828912 ip-10-0-131-126 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 26 16:00:47.830852 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.830750 2567 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" May 26 16:00:47.834228 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834213 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:47.834228 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834228 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834233 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834237 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834240 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834242 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834246 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834248 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834251 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834254 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834257 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834265 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834270 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834274 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834278 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834280 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834283 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834285 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834288 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834291 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:47.834286 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834294 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834297 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834299 2567 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834302 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834305 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834308 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834310 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834312 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834317 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834321 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834324 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834326 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834329 2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834332 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834334 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834337 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834340 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834343 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834345 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:47.834730 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834348 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834351 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834353 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834356 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834358 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834360 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834363 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834365 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834368 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834370 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834373 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834375 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834378 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834380 2567 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834383 2567 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834386 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834389 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834392 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834394 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834397 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:47.835248 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834399 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834402 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834404 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834407 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834410 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834413 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834416 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834419 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834421 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834423 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834426 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834429 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834432 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834435 2567 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834437 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834441 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834443 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834446 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834449 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834451 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:47.835733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834454 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834457 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834459 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834462 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834464 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834467 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834469 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834864 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834871 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834875 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834879 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834882 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834886 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834888 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834891 2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834894 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834896 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834900 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834903 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834905 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:47.836230 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834908 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834910 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834913 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834915 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834918 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834921 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834923 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834926 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834929 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834944 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834946 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834949 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834953 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834956 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834958 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834961 2567 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834964 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834966 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834969 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834972 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:47.836702 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834975 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834977 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834980 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834982 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834985 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834987 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834991 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834993 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834996 2567 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.834998 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835002 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835005 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835007 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835010 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835012 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835015 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835018 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835020 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835023 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835026 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:47.837212 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835029 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835032 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835034 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835037 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835039 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835042 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835044 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835047 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835049 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835052 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835055 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835059 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835063 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835065 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835068 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835070 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835073 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835075 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835078 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:47.837681 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835080 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835083 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835085 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835088 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835091 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835093 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835096 2567 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835098 2567 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835101 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835103 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835106 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835108 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835111 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.835113 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836073 2567 flags.go:64] FLAG: --address="0.0.0.0" May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836083 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836093 2567 flags.go:64] FLAG: --anonymous-auth="true" May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836098 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836103 2567 flags.go:64] FLAG: --authentication-token-webhook="false" May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836106 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836111 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" May 26 16:00:47.838153 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836115 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836118 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836121 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836125 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836129 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836132 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836135 2567 flags.go:64] FLAG: --cgroup-root="" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836138 2567 flags.go:64] FLAG: --cgroups-per-qos="true" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836141 2567 flags.go:64] FLAG: --client-ca-file="" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836144 2567 flags.go:64] FLAG: --cloud-config="" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836147 2567 flags.go:64] FLAG: --cloud-provider="external" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836151 2567 flags.go:64] FLAG: --cluster-dns="[]" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836155 2567 flags.go:64] FLAG: --cluster-domain="" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836158 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836162 2567 flags.go:64] FLAG: --config-dir="" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836165 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836169 2567 flags.go:64] FLAG: --container-log-max-files="5" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836173 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836176 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836179 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836183 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836186 2567 flags.go:64] FLAG: --contention-profiling="false" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836189 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836192 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836195 2567 flags.go:64] FLAG: --cpu-manager-policy="none" May 26 16:00:47.838663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836199 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836203 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836206 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836209 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836212 2567 flags.go:64] FLAG: --enable-load-reader="false" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836215 2567 flags.go:64] FLAG: --enable-server="true" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836218 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836223 2567 flags.go:64] FLAG: --event-burst="100" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836227 2567 flags.go:64] FLAG: --event-qps="50" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836230 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836233 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836236 2567 flags.go:64] FLAG: --eviction-hard="" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836240 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836244 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836247 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836250 2567 flags.go:64] FLAG: --eviction-soft="" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836253 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836256 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836258 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836261 2567 flags.go:64] FLAG: --experimental-mounter-path="" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836264 2567 flags.go:64] FLAG: --fail-cgroupv1="false" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836267 2567 flags.go:64] FLAG: --fail-swap-on="true" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836270 2567 flags.go:64] FLAG: --feature-gates="" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836274 2567 flags.go:64] FLAG: --file-check-frequency="20s" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836278 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" May 26 16:00:47.839318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836281 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836284 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836288 2567 flags.go:64] FLAG: --healthz-port="10248" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836291 2567 flags.go:64] FLAG: --help="false" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836294 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-131-126.ec2.internal" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836297 2567 flags.go:64] FLAG: --housekeeping-interval="10s" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836300 2567 flags.go:64] FLAG: --http-check-frequency="20s" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836302 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836306 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836309 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836312 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836315 2567 flags.go:64] FLAG: --image-service-endpoint="" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836318 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836321 2567 flags.go:64] FLAG: --kube-api-burst="100" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836324 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836327 2567 flags.go:64] FLAG: --kube-api-qps="50" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836330 2567 flags.go:64] FLAG: --kube-reserved="" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836333 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836336 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836339 2567 flags.go:64] FLAG: --kubelet-cgroups="" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836342 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836345 2567 flags.go:64] FLAG: --lock-file="" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836348 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836351 2567 flags.go:64] FLAG: --log-flush-frequency="5s" May 26 16:00:47.839914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836354 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836359 2567 flags.go:64] FLAG: --log-json-split-stream="false" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836362 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836365 2567 flags.go:64] FLAG: --log-text-split-stream="false" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836368 2567 flags.go:64] FLAG: --logging-format="text" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836371 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836374 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836377 2567 flags.go:64] FLAG: --manifest-url="" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836380 2567 flags.go:64] FLAG: --manifest-url-header="" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836385 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836389 2567 flags.go:64] FLAG: --max-open-files="1000000" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836393 2567 flags.go:64] FLAG: --max-pods="110" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836396 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836399 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836402 2567 flags.go:64] FLAG: --memory-manager-policy="None" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836405 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836408 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836412 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836415 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836422 2567 flags.go:64] FLAG: --node-status-max-images="50" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836425 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836428 2567 flags.go:64] FLAG: --oom-score-adj="-999" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836431 2567 flags.go:64] FLAG: --pod-cidr="" May 26 16:00:47.840552 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836435 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836441 2567 flags.go:64] FLAG: --pod-manifest-path="" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836444 2567 flags.go:64] FLAG: --pod-max-pids="-1" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836447 2567 flags.go:64] FLAG: --pods-per-core="0" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836453 2567 flags.go:64] FLAG: --port="10250" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836456 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836459 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02eec544a5d0562b1" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836462 2567 flags.go:64] FLAG: --qos-reserved="" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836465 2567 flags.go:64] FLAG: --read-only-port="10255" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836468 2567 flags.go:64] FLAG: --register-node="true" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836471 2567 flags.go:64] FLAG: --register-schedulable="true" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836474 2567 flags.go:64] FLAG: --register-with-taints="" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836478 2567 flags.go:64] FLAG: --registry-burst="10" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836481 2567 flags.go:64] FLAG: --registry-qps="5" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836484 2567 flags.go:64] FLAG: --reserved-cpus="" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836487 2567 flags.go:64] FLAG: --reserved-memory="" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836490 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836494 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836497 2567 flags.go:64] FLAG: --rotate-certificates="false" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836500 2567 flags.go:64] FLAG: --rotate-server-certificates="false" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836503 2567 flags.go:64] FLAG: --runonce="false" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836507 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836510 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836514 2567 flags.go:64] FLAG: --seccomp-default="false" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836516 2567 flags.go:64] FLAG: --serialize-image-pulls="true" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836519 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" May 26 16:00:47.841115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836522 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836526 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836529 2567 flags.go:64] FLAG: --storage-driver-password="root" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836532 2567 flags.go:64] FLAG: --storage-driver-secure="false" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836535 2567 flags.go:64] FLAG: --storage-driver-table="stats" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836538 2567 flags.go:64] FLAG: --storage-driver-user="root" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836541 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836544 2567 flags.go:64] FLAG: --sync-frequency="1m0s" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836547 2567 flags.go:64] FLAG: --system-cgroups="" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836550 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836558 2567 flags.go:64] FLAG: --system-reserved-cgroup="" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836561 2567 flags.go:64] FLAG: --tls-cert-file="" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836564 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836568 2567 flags.go:64] FLAG: --tls-min-version="" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836571 2567 flags.go:64] FLAG: --tls-private-key-file="" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836573 2567 flags.go:64] FLAG: --topology-manager-policy="none" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836576 2567 flags.go:64] FLAG: --topology-manager-policy-options="" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836580 2567 flags.go:64] FLAG: --topology-manager-scope="container" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836583 2567 flags.go:64] FLAG: --v="2" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836587 2567 flags.go:64] FLAG: --version="false" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836591 2567 flags.go:64] FLAG: --vmodule="" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836596 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.836599 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836688 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:47.841731 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836691 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836696 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836699 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836702 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836705 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836707 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836710 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836713 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836715 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836718 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836721 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836724 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836726 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836729 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836732 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836734 2567 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836737 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836739 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836743 2567 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836745 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:47.842341 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836748 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836750 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836753 2567 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836755 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836758 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836760 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836763 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836766 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836768 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836771 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836773 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836777 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836780 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836784 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836787 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836789 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836792 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836795 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836797 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:47.842879 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836800 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836802 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836805 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836807 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836810 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836812 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836815 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836817 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836819 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836822 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836825 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836828 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836830 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836833 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836835 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836838 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836841 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836843 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836846 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:47.843399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836848 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836852 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836856 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836859 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836862 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836864 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836867 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836870 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836872 2567 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836875 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836878 2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836880 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836883 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836885 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836887 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836890 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836893 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836895 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836898 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836918 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:47.843880 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836922 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836925 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836927 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836930 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836945 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836947 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.836950 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.838013 2567 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.844264 2567 server.go:530] "Kubelet version" kubeletVersion="v1.33.11" May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.844280 2567 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844328 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844333 2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844338 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844342 2567 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844345 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844348 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:47.844387 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844351 2567 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844354 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844357 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844360 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844363 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844367 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844370 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844373 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844375 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844378 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844381 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844383 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844386 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844389 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844393 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844396 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844400 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844402 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844405 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844408 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:47.844842 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844410 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844413 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844415 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844418 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844420 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844423 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844426 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844429 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844431 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844434 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844437 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844440 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844442 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844445 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844448 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844450 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844453 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844455 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844462 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844465 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:47.845353 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844468 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844470 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844474 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844478 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844482 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844485 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844488 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844491 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844494 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844497 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844500 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844502 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844505 2567 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844508 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844510 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844513 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844516 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844519 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844522 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:47.845883 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844524 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844527 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844530 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844533 2567 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844536 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844539 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844541 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844544 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844547 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844549 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844552 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844554 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844558 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844560 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844563 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844565 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844568 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844570 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844573 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:47.846377 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844575 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844578 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.844583 2567 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844682 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844687 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844690 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844692 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844695 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844698 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844700 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844703 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844706 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844709 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844711 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844714 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:47.846848 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844716 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844719 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844722 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844725 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844727 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844732 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844736 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844740 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844743 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844746 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844750 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844754 2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844757 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844760 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844763 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844766 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844768 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844771 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844774 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:47.847245 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844776 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844779 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844781 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844784 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844786 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844789 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844791 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844794 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844797 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844800 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844803 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844805 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844808 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844810 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844813 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844816 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844818 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844821 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844823 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844826 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:47.847915 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844829 2567 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844831 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844834 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844836 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844839 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844842 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844845 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844848 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844850 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844853 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844856 2567 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844858 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844861 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844863 2567 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844866 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844868 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844870 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844873 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844875 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844878 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:47.848450 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844880 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844883 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844886 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844888 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844891 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844893 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844896 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844898 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844901 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844904 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844906 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844909 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844911 2567 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844914 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:47.844917 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.844921 2567 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 26 16:00:47.849022 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.846268 2567 server.go:962] "Client rotation is on, will bootstrap in background" May 26 16:00:47.850159 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.850142 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" May 26 16:00:47.851129 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.851118 2567 server.go:1019] "Starting client certificate rotation" May 26 16:00:47.851232 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.851216 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" May 26 16:00:47.851265 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.851257 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" May 26 16:00:47.881274 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.881256 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 26 16:00:47.883710 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.883692 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 26 16:00:47.903023 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.903004 2567 log.go:25] "Validated CRI v1 runtime API" May 26 16:00:47.908811 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.908797 2567 log.go:25] "Validated CRI v1 image API" May 26 16:00:47.909968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.909952 2567 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" May 26 16:00:47.913018 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.913000 2567 fs.go:135] Filesystem UUIDs: map[295bbb30-83df-4aaf-9f80-9cfb5ee1a75f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cb46dade-4dd6-4622-af86-f6ed35b425a7:/dev/nvme0n1p4] May 26 16:00:47.913079 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.913018 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] May 26 16:00:47.914192 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.914175 2567 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 26 16:00:47.918566 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.918068 2567 manager.go:217] Machine: {Timestamp:2026-05-26 16:00:47.91703155 +0000 UTC m=+0.479764483 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099861 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec292b7d1b4161785aa5e408b574ba81 SystemUUID:ec292b7d-1b41-6178-5aa5-e408b574ba81 BootID:e1cd8e6b-9d45-4ad6-845a-ed769a9f9e56 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:c3:4c:5d:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:c3:4c:5d:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:7a:9f:c8:48:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} May 26 16:00:47.918566 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.918339 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. May 26 16:00:47.918566 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.918514 2567 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} May 26 16:00:47.922262 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.922236 2567 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] May 26 16:00:47.922391 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.922263 2567 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-126.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} May 26 16:00:47.922432 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.922402 2567 topology_manager.go:138] "Creating topology manager with none policy" May 26 16:00:47.922432 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.922410 2567 container_manager_linux.go:306] "Creating device plugin manager" May 26 16:00:47.922432 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.922418 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" May 26 16:00:47.924568 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.924557 2567 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" May 26 16:00:47.926479 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.926469 2567 state_mem.go:36] "Initialized new in-memory state store" May 26 16:00:47.926583 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.926575 2567 server.go:1267] "Using root directory" path="/var/lib/kubelet" May 26 16:00:47.930230 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.930220 2567 kubelet.go:491] "Attempting to sync node with API server" May 26 16:00:47.930269 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.930234 2567 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" May 26 16:00:47.930269 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.930253 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" May 26 16:00:47.930269 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.930266 2567 kubelet.go:397] "Adding apiserver pod source" May 26 16:00:47.930356 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.930274 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" May 26 16:00:47.931374 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.931362 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 26 16:00:47.931424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.931381 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 26 16:00:47.934092 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.934071 2567 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1" May 26 16:00:47.936010 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.935996 2567 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" May 26 16:00:47.936913 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936902 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" May 26 16:00:47.936968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936919 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" May 26 16:00:47.936968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936925 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" May 26 16:00:47.936968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936943 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" May 26 16:00:47.936968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936952 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" May 26 16:00:47.936968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936958 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" May 26 16:00:47.936968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936964 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" May 26 16:00:47.936968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936970 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" May 26 16:00:47.937172 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936977 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" May 26 16:00:47.937172 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936982 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" May 26 16:00:47.937172 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.936991 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" May 26 16:00:47.937459 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.937450 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" May 26 16:00:47.938250 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.938239 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" May 26 16:00:47.938284 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.938254 2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" May 26 16:00:47.941690 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:47.941662 2567 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" May 26 16:00:47.941768 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:47.941662 2567 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" May 26 16:00:47.942423 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.942410 2567 watchdog_linux.go:99] "Systemd watchdog is not enabled" May 26 16:00:47.942454 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.942448 2567 server.go:1295] "Started kubelet" May 26 16:00:47.942537 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.942504 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 May 26 16:00:47.942620 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.942538 2567 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 May 26 16:00:47.942620 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.942613 2567 server_v1.go:47] "podresources" method="list" useActivePods=true May 26 16:00:47.943323 ip-10-0-131-126 systemd[1]: Started Kubernetes Kubelet. May 26 16:00:47.944013 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.943999 2567 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" May 26 16:00:47.945885 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.945872 2567 server.go:317] "Adding debug handlers to kubelet server" May 26 16:00:47.950702 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.950687 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" May 26 16:00:47.950790 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.950763 2567 volume_manager.go:295] "The desired_state_of_world populator starts" May 26 16:00:47.950790 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.950775 2567 volume_manager.go:297] "Starting Kubelet Volume Manager" May 26 16:00:47.950904 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.950862 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" May 26 16:00:47.950904 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.950890 2567 desired_state_of_world_populator.go:150] "Desired state populator starts to run" May 26 16:00:47.951020 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.950964 2567 reconstruct.go:97] "Volume reconstruction finished" May 26 16:00:47.951020 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.950972 2567 reconciler.go:26] "Reconciler: start to sync state" May 26 16:00:47.951880 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:47.951701 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:47.952455 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.952437 2567 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-126.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope May 26 16:00:47.952837 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.952821 2567 factory.go:55] Registering systemd factory May 26 16:00:47.952956 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.952929 2567 factory.go:223] Registration of the systemd container factory successfully May 26 16:00:47.952956 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:47.952864 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" May 26 16:00:47.953596 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:47.952495 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-126.ec2.internal.18b328e670da52f7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-126.ec2.internal,UID:ip-10-0-131-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-126.ec2.internal,},FirstTimestamp:2026-05-26 16:00:47.942423287 +0000 UTC m=+0.505156221,LastTimestamp:2026-05-26 16:00:47.942423287 +0000 UTC m=+0.505156221,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-126.ec2.internal,}" May 26 16:00:47.953752 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.953717 2567 factory.go:153] Registering CRI-O factory May 26 16:00:47.953824 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.953790 2567 factory.go:223] Registration of the crio container factory successfully May 26 16:00:47.953894 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.953880 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory May 26 16:00:47.954000 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.953988 2567 factory.go:103] Registering Raw factory May 26 16:00:47.954042 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.954035 2567 manager.go:1196] Started watching for new ooms in manager May 26 16:00:47.954992 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.954977 2567 manager.go:319] Starting recovery of all containers May 26 16:00:47.956255 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:47.956213 2567 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" May 26 16:00:47.956355 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.956303 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d5jnk" May 26 16:00:47.957079 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:47.957056 2567 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" May 26 16:00:47.962615 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.962591 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d5jnk" May 26 16:00:47.966259 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.966246 2567 manager.go:324] Recovery completed May 26 16:00:47.970312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.970300 2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:47.972000 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.971986 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:47.972062 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.972014 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:47.972062 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.972031 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:47.972515 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.972502 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" May 26 16:00:47.972515 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.972514 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" May 26 16:00:47.972628 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.972533 2567 state_mem.go:36] "Initialized new in-memory state store" May 26 16:00:47.973813 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:47.973749 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-126.ec2.internal.18b328e6729d9ed3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-126.ec2.internal,UID:ip-10-0-131-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-126.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-126.ec2.internal,},FirstTimestamp:2026-05-26 16:00:47.971999443 +0000 UTC m=+0.534732376,LastTimestamp:2026-05-26 16:00:47.971999443 +0000 UTC m=+0.534732376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-126.ec2.internal,}" May 26 16:00:47.975961 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.975948 2567 policy_none.go:49] "None policy: Start" May 26 16:00:47.976036 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.975965 2567 memory_manager.go:186] "Starting memorymanager" policy="None" May 26 16:00:47.976444 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:47.976435 2567 state_mem.go:35] "Initializing new in-memory state store" May 26 16:00:48.019425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.018458 2567 manager.go:341] "Starting Device Plugin manager" May 26 16:00:48.019425 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.018555 2567 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" May 26 16:00:48.019425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.018566 2567 server.go:85] "Starting device plugin registration server" May 26 16:00:48.019425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.018766 2567 eviction_manager.go:189] "Eviction manager: starting control loop" May 26 16:00:48.019425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.018774 2567 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" May 26 16:00:48.019425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.018875 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" May 26 16:00:48.019425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.018962 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" May 26 16:00:48.019425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.018971 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" May 26 16:00:48.019717 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.019499 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" May 26 16:00:48.019717 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.019538 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.043701 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.043672 2567 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" May 26 16:00:48.044929 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.044908 2567 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" May 26 16:00:48.044929 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.044947 2567 status_manager.go:230] "Starting to sync pod status with apiserver" May 26 16:00:48.045079 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.044967 2567 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." May 26 16:00:48.045079 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.044974 2567 kubelet.go:2451] "Starting kubelet main sync loop" May 26 16:00:48.045079 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.045018 2567 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" May 26 16:00:48.048205 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.048189 2567 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:48.119422 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.119399 2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:48.126343 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.126315 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:48.126343 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.126348 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:48.126494 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.126364 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:48.126494 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.126387 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-126.ec2.internal" May 26 16:00:48.135592 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.135573 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-126.ec2.internal" May 26 16:00:48.135646 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.135595 2567 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-126.ec2.internal\": node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.145315 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.145295 2567 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal"] May 26 16:00:48.145371 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.145357 2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:48.146097 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.146075 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:48.146190 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.146106 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:48.146190 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.146118 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:48.146615 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.146599 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.147260 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.147248 2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:48.147400 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.147386 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" May 26 16:00:48.147478 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.147417 2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:48.147922 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.147905 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:48.148005 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.147948 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:48.148005 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.147966 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:48.148005 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.147983 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:48.148005 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.147989 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:48.148005 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.148001 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:48.149045 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.149026 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal" May 26 16:00:48.149122 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.149055 2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:48.149744 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.149728 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:48.149809 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.149756 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:48.149809 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.149767 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:48.184566 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.184543 2567 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-126.ec2.internal\" not found" node="ip-10-0-131-126.ec2.internal" May 26 16:00:48.188880 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.188863 2567 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-126.ec2.internal\" not found" node="ip-10-0-131-126.ec2.internal" May 26 16:00:48.246795 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.246770 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.251891 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.251873 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86e0aea22240a288454e3db6d47c9448-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal\" (UID: \"86e0aea22240a288454e3db6d47c9448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" May 26 16:00:48.251973 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.251900 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e0aea22240a288454e3db6d47c9448-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal\" (UID: \"86e0aea22240a288454e3db6d47c9448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" May 26 16:00:48.251973 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.251921 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f187ab729b0a5371eb7618f4a50c541-config\") pod \"kube-apiserver-proxy-ip-10-0-131-126.ec2.internal\" (UID: \"4f187ab729b0a5371eb7618f4a50c541\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal" May 26 16:00:48.347337 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.347291 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.352599 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.352581 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86e0aea22240a288454e3db6d47c9448-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal\" (UID: \"86e0aea22240a288454e3db6d47c9448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" May 26 16:00:48.352699 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.352614 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e0aea22240a288454e3db6d47c9448-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal\" (UID: \"86e0aea22240a288454e3db6d47c9448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" May 26 16:00:48.352699 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.352640 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f187ab729b0a5371eb7618f4a50c541-config\") pod \"kube-apiserver-proxy-ip-10-0-131-126.ec2.internal\" (UID: \"4f187ab729b0a5371eb7618f4a50c541\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal" May 26 16:00:48.352699 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.352669 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86e0aea22240a288454e3db6d47c9448-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal\" (UID: \"86e0aea22240a288454e3db6d47c9448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" May 26 16:00:48.352792 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.352673 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f187ab729b0a5371eb7618f4a50c541-config\") pod \"kube-apiserver-proxy-ip-10-0-131-126.ec2.internal\" (UID: \"4f187ab729b0a5371eb7618f4a50c541\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal" May 26 16:00:48.352792 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.352675 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e0aea22240a288454e3db6d47c9448-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal\" (UID: \"86e0aea22240a288454e3db6d47c9448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" May 26 16:00:48.448066 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.447983 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.487432 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.487408 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" May 26 16:00:48.491223 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.491205 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal" May 26 16:00:48.548359 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.548315 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.648881 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.648835 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.749423 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.749333 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.849973 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.849924 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.851040 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.851029 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" May 26 16:00:48.851192 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.851169 2567 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 26 16:00:48.950857 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:48.950829 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:48.951944 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.951915 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" May 26 16:00:48.964991 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.964959 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 15:55:47 +0000 UTC" deadline="2027-11-25 13:10:24.849286065 +0000 UTC" May 26 16:00:48.964991 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.964987 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13149h9m35.884302433s" May 26 16:00:48.972092 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.972072 2567 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 26 16:00:48.991060 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.991039 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jj8th" May 26 16:00:48.998590 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:48.998573 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jj8th" May 26 16:00:49.008111 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:49.008051 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e0aea22240a288454e3db6d47c9448.slice/crio-db2834d9a925d6fac4b2831a9d2626b815cc251418739c06accd99842b9ca673 WatchSource:0}: Error finding container db2834d9a925d6fac4b2831a9d2626b815cc251418739c06accd99842b9ca673: Status 404 returned error can't find the container with id db2834d9a925d6fac4b2831a9d2626b815cc251418739c06accd99842b9ca673 May 26 16:00:49.008320 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:49.008301 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f187ab729b0a5371eb7618f4a50c541.slice/crio-91fbd28797fbb39a8daa68b0e41ecc60fef73aae19617a70bec124eeed57ba09 WatchSource:0}: Error finding container 91fbd28797fbb39a8daa68b0e41ecc60fef73aae19617a70bec124eeed57ba09: Status 404 returned error can't find the container with id 91fbd28797fbb39a8daa68b0e41ecc60fef73aae19617a70bec124eeed57ba09 May 26 16:00:49.011426 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.011406 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:00:49.049797 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.049744 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal" event={"ID":"4f187ab729b0a5371eb7618f4a50c541","Type":"ContainerStarted","Data":"91fbd28797fbb39a8daa68b0e41ecc60fef73aae19617a70bec124eeed57ba09"} May 26 16:00:49.050584 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.050560 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" event={"ID":"86e0aea22240a288454e3db6d47c9448","Type":"ContainerStarted","Data":"db2834d9a925d6fac4b2831a9d2626b815cc251418739c06accd99842b9ca673"} May 26 16:00:49.051660 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:49.051645 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:49.152150 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:49.152116 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:49.186173 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.186149 2567 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:49.253016 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:49.252988 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:49.353481 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:49.353444 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:49.454306 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:49.454272 2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-126.ec2.internal\" not found" May 26 16:00:49.487806 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.487776 2567 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:49.532684 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.532610 2567 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:49.551620 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.551521 2567 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" May 26 16:00:49.562562 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.562459 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 26 16:00:49.563578 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.563558 2567 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal" May 26 16:00:49.572467 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.572416 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 26 16:00:49.931417 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.931379 2567 apiserver.go:52] "Watching apiserver" May 26 16:00:49.940704 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.940679 2567 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" May 26 16:00:49.941784 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.941756 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal","openshift-image-registry/node-ca-q7xm7","openshift-multus/multus-additional-cni-plugins-z2msh","openshift-multus/multus-l6rc8","openshift-multus/network-metrics-daemon-x7p82","openshift-network-operator/iptables-alerter-p72d6","kube-system/global-pull-secret-syncer-6nfnx","kube-system/konnectivity-agent-qg7wc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp","openshift-cluster-node-tuning-operator/tuned-t495h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal","openshift-network-diagnostics/network-check-target-r6sp2","openshift-ovn-kubernetes/ovnkube-node-vflns"] May 26 16:00:49.944252 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.944230 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.944460 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.944416 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:49.945625 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.945607 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:49.946628 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.946603 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k5f4l\"" May 26 16:00:49.946628 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.946614 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" May 26 16:00:49.946776 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.946737 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8tnc4\"" May 26 16:00:49.946841 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.946793 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" May 26 16:00:49.946841 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.946795 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" May 26 16:00:49.946954 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.946899 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" May 26 16:00:49.947062 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.947047 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" May 26 16:00:49.947702 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.947686 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6d4qv\"" May 26 16:00:49.947702 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.947696 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" May 26 16:00:49.947946 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.947907 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l6rc8" May 26 16:00:49.948079 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.948053 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:49.948167 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:49.948123 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:00:49.948554 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.948374 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" May 26 16:00:49.948554 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.948436 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" May 26 16:00:49.948554 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.948472 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" May 26 16:00:49.948554 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.948521 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" May 26 16:00:49.949131 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.949101 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:49.949753 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.949735 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" May 26 16:00:49.949889 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.949871 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kfhvp\"" May 26 16:00:49.950250 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.950221 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qg7wc" May 26 16:00:49.951086 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.951065 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" May 26 16:00:49.951180 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.951099 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" May 26 16:00:49.952500 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.951579 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" May 26 16:00:49.952500 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.951814 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dr2pn\"" May 26 16:00:49.952500 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.952419 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p72cf\"" May 26 16:00:49.952500 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.952486 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" May 26 16:00:49.952727 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.952536 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" May 26 16:00:49.954584 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.954497 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:49.956254 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.956238 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:49.956349 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:49.956297 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:00:49.956417 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.956364 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:49.956634 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.956618 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h8xtp\"" May 26 16:00:49.956778 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.956744 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" May 26 16:00:49.956963 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.956918 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" May 26 16:00:49.956963 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.956950 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" May 26 16:00:49.957723 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.957702 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:49.957804 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:49.957771 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:00:49.958740 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.958720 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" May 26 16:00:49.958835 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.958725 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" May 26 16:00:49.958835 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.958790 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qz8wz\"" May 26 16:00:49.959738 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.959721 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" May 26 16:00:49.959822 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.959770 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" May 26 16:00:49.959822 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.959811 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" May 26 16:00:49.959947 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.959846 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" May 26 16:00:49.961285 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961265 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d21ccc94-60a4-4663-bbf1-a243a95bfa44-konnectivity-ca\") pod \"konnectivity-agent-qg7wc\" (UID: \"d21ccc94-60a4-4663-bbf1-a243a95bfa44\") " pod="kube-system/konnectivity-agent-qg7wc" May 26 16:00:49.961360 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961301 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-system-cni-dir\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.961360 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961329 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-device-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:49.961465 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961353 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:49.961465 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961389 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-kubernetes\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.961465 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961416 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-run\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.961465 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961443 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-var-lib-cni-bin\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.961652 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f256df7-c5ad-4a87-8369-dace4f5d89d1-host-slash\") pod \"iptables-alerter-p72d6\" (UID: \"3f256df7-c5ad-4a87-8369-dace4f5d89d1\") " pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:49.961652 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961532 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-sysctl-d\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.961652 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961583 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-sysctl-conf\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.961652 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961627 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nktq\" (UniqueName: \"kubernetes.io/projected/77d4a48a-9687-4338-b49d-1219168b52e2-kube-api-access-7nktq\") pod \"node-ca-q7xm7\" (UID: \"77d4a48a-9687-4338-b49d-1219168b52e2\") " pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:49.961825 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961660 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-run-k8s-cni-cncf-io\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.961825 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961703 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-socket-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:49.961825 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961726 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-lib-modules\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.961825 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/442af588-81ae-4c16-bafd-6a466acbaf11-cni-binary-copy\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:49.961825 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961777 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77d4a48a-9687-4338-b49d-1219168b52e2-host\") pod \"node-ca-q7xm7\" (UID: \"77d4a48a-9687-4338-b49d-1219168b52e2\") " pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:49.961825 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961801 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:49.962041 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961843 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxxz\" (UniqueName: \"kubernetes.io/projected/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-kube-api-access-9bxxz\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962041 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961886 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-sys\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.962041 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961921 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-cnibin\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:49.962041 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.961979 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-cni-binary-copy\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962041 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962002 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-registration-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:49.962041 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962020 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-system-cni-dir\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:49.962041 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962036 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/442af588-81ae-4c16-bafd-6a466acbaf11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962059 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962082 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-tuned\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962106 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6r9\" (UniqueName: \"kubernetes.io/projected/3f256df7-c5ad-4a87-8369-dace4f5d89d1-kube-api-access-sf6r9\") pod \"iptables-alerter-p72d6\" (UID: \"3f256df7-c5ad-4a87-8369-dace4f5d89d1\") " pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962141 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4fd\" (UniqueName: \"kubernetes.io/projected/ddf6bd23-ce8a-4927-b11f-c552d481bc33-kube-api-access-wb4fd\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962169 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/442af588-81ae-4c16-bafd-6a466acbaf11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962194 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-cnibin\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962215 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-sysconfig\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962254 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wh7\" (UniqueName: \"kubernetes.io/projected/442af588-81ae-4c16-bafd-6a466acbaf11-kube-api-access-49wh7\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962279 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-cni-dir\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962321 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f256df7-c5ad-4a87-8369-dace4f5d89d1-iptables-alerter-script\") pod \"iptables-alerter-p72d6\" (UID: \"3f256df7-c5ad-4a87-8369-dace4f5d89d1\") " pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962344 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-host\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962368 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-run-multus-certs\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-sys-fs\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962413 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-tmp\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-socket-dir-parent\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962512 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-run-netns\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962539 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-var-lib-kubelet\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962562 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-etc-kubernetes\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962587 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzs9f\" (UniqueName: \"kubernetes.io/projected/2784dfc5-43c4-434e-be4d-5698f348de7a-kube-api-access-dzs9f\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962627 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-os-release\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962648 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-hostroot\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962682 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-modprobe-d\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962707 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-var-lib-kubelet\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962729 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:49.962766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962765 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d21ccc94-60a4-4663-bbf1-a243a95bfa44-agent-certs\") pod \"konnectivity-agent-qg7wc\" (UID: \"d21ccc94-60a4-4663-bbf1-a243a95bfa44\") " pod="kube-system/konnectivity-agent-qg7wc" May 26 16:00:49.963349 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962810 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-var-lib-cni-multus\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.963349 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962839 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-daemon-config\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.963349 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962873 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9n7h\" (UniqueName: \"kubernetes.io/projected/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-kube-api-access-l9n7h\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.963349 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962897 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77d4a48a-9687-4338-b49d-1219168b52e2-serviceca\") pod \"node-ca-q7xm7\" (UID: \"77d4a48a-9687-4338-b49d-1219168b52e2\") " pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:49.963349 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962964 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-os-release\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:49.963349 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.962996 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-conf-dir\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:49.963349 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.963017 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-systemd\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:49.999337 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.999308 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:48 +0000 UTC" deadline="2027-10-31 01:15:33.011052554 +0000 UTC" May 26 16:00:49.999337 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:49.999333 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12537h14m43.011722106s" May 26 16:00:50.052210 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.052138 2567 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" May 26 16:00:50.063980 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.063949 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-node-log\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.064136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.063988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:50.064136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064021 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/442af588-81ae-4c16-bafd-6a466acbaf11-cni-binary-copy\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.064136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064046 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-run-systemd\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.064136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064071 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-run-openvswitch\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.064136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.064136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064112 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77d4a48a-9687-4338-b49d-1219168b52e2-host\") pod \"node-ca-q7xm7\" (UID: \"77d4a48a-9687-4338-b49d-1219168b52e2\") " pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:50.064136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064127 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064148 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxxz\" (UniqueName: \"kubernetes.io/projected/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-kube-api-access-9bxxz\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064171 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-sys\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064195 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-var-lib-openvswitch\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3874d0e8-75c4-4fed-be1a-f305a44abfa8-dbus\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064244 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-cnibin\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064263 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-cni-binary-copy\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064278 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-registration-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064294 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-run-ovn-kubernetes\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-cni-bin\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064333 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-system-cni-dir\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064357 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/442af588-81ae-4c16-bafd-6a466acbaf11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064383 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064406 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-tuned\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064428 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-ovn-node-metrics-cert\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064445 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6r9\" (UniqueName: \"kubernetes.io/projected/3f256df7-c5ad-4a87-8369-dace4f5d89d1-kube-api-access-sf6r9\") pod \"iptables-alerter-p72d6\" (UID: \"3f256df7-c5ad-4a87-8369-dace4f5d89d1\") " pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:50.064559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064461 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4fd\" (UniqueName: \"kubernetes.io/projected/ddf6bd23-ce8a-4927-b11f-c552d481bc33-kube-api-access-wb4fd\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064486 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/442af588-81ae-4c16-bafd-6a466acbaf11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064512 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-cnibin\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064536 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-sysconfig\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064561 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-kubelet\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064585 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-run-netns\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064603 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-sys\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064611 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-system-cni-dir\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064617 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/442af588-81ae-4c16-bafd-6a466acbaf11-cni-binary-copy\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064613 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-cni-netd\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064687 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-cnibin\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064694 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77d4a48a-9687-4338-b49d-1219168b52e2-host\") pod \"node-ca-q7xm7\" (UID: \"77d4a48a-9687-4338-b49d-1219168b52e2\") " pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064798 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-registration-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064821 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064839 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-env-overrides\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064883 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-49wh7\" (UniqueName: \"kubernetes.io/projected/442af588-81ae-4c16-bafd-6a466acbaf11-kube-api-access-49wh7\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064923 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-cni-dir\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.065303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064963 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f256df7-c5ad-4a87-8369-dace4f5d89d1-iptables-alerter-script\") pod \"iptables-alerter-p72d6\" (UID: \"3f256df7-c5ad-4a87-8369-dace4f5d89d1\") " pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.064988 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-host\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065029 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-run-multus-certs\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065054 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-sys-fs\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065078 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-tmp\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065105 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r45ll\" (UniqueName: \"kubernetes.io/projected/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-kube-api-access-r45ll\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065132 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-socket-dir-parent\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065158 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-run-netns\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065228 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-var-lib-kubelet\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065253 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-etc-kubernetes\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065281 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dzs9f\" (UniqueName: \"kubernetes.io/projected/2784dfc5-43c4-434e-be4d-5698f348de7a-kube-api-access-dzs9f\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065299 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-cni-binary-copy\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065304 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/442af588-81ae-4c16-bafd-6a466acbaf11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065305 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/442af588-81ae-4c16-bafd-6a466acbaf11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065341 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-etc-openvswitch\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065343 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-run-multus-certs\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065367 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-sys-fs\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.066061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065372 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-sysconfig\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065371 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-os-release\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065372 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-socket-dir-parent\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065388 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-run-netns\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065434 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-os-release\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065446 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-hostroot\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065448 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-var-lib-kubelet\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065472 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-etc-kubernetes\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065473 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-modprobe-d\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065503 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-var-lib-kubelet\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065510 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-cni-dir\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065523 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065548 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-hostroot\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065552 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065568 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-var-lib-kubelet\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065578 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d21ccc94-60a4-4663-bbf1-a243a95bfa44-agent-certs\") pod \"konnectivity-agent-qg7wc\" (UID: \"d21ccc94-60a4-4663-bbf1-a243a95bfa44\") " pod="kube-system/konnectivity-agent-qg7wc" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065592 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-host\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065619 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" May 26 16:00:50.066713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065631 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-modprobe-d\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065669 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-cnibin\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065671 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-var-lib-cni-multus\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065624 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-var-lib-cni-multus\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065722 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-daemon-config\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.065727 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065746 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l9n7h\" (UniqueName: \"kubernetes.io/projected/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-kube-api-access-l9n7h\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065767 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77d4a48a-9687-4338-b49d-1219168b52e2-serviceca\") pod \"node-ca-q7xm7\" (UID: \"77d4a48a-9687-4338-b49d-1219168b52e2\") " pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.065802 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs podName:ddf6bd23-ce8a-4927-b11f-c552d481bc33 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:50.565768205 +0000 UTC m=+3.128501142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs") pod "network-metrics-daemon-x7p82" (UID: "ddf6bd23-ce8a-4927-b11f-c552d481bc33") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065832 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-systemd-units\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065859 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-run-ovn\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065887 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-os-release\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065913 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-conf-dir\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065956 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-systemd\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.065985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3874d0e8-75c4-4fed-be1a-f305a44abfa8-kubelet-config\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066013 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d21ccc94-60a4-4663-bbf1-a243a95bfa44-konnectivity-ca\") pod \"konnectivity-agent-qg7wc\" (UID: \"d21ccc94-60a4-4663-bbf1-a243a95bfa44\") " pod="kube-system/konnectivity-agent-qg7wc" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066037 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-system-cni-dir\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.067534 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066063 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-device-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066088 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066109 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f256df7-c5ad-4a87-8369-dace4f5d89d1-iptables-alerter-script\") pod \"iptables-alerter-p72d6\" (UID: \"3f256df7-c5ad-4a87-8369-dace4f5d89d1\") " pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066111 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-kubernetes\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066160 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-kubernetes\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066188 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77d4a48a-9687-4338-b49d-1219168b52e2-serviceca\") pod \"node-ca-q7xm7\" (UID: \"77d4a48a-9687-4338-b49d-1219168b52e2\") " pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066262 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/442af588-81ae-4c16-bafd-6a466acbaf11-os-release\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066284 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-daemon-config\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066300 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-system-cni-dir\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066314 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-device-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066368 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-systemd\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066367 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066405 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-multus-conf-dir\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066419 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-run\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066452 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-log-socket\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066483 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-ovnkube-config\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066484 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-run\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066528 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-var-lib-cni-bin\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.068309 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066570 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-var-lib-cni-bin\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066563 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f256df7-c5ad-4a87-8369-dace4f5d89d1-host-slash\") pod \"iptables-alerter-p72d6\" (UID: \"3f256df7-c5ad-4a87-8369-dace4f5d89d1\") " pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066607 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-sysctl-d\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066618 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f256df7-c5ad-4a87-8369-dace4f5d89d1-host-slash\") pod \"iptables-alerter-p72d6\" (UID: \"3f256df7-c5ad-4a87-8369-dace4f5d89d1\") " pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066629 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d21ccc94-60a4-4663-bbf1-a243a95bfa44-konnectivity-ca\") pod \"konnectivity-agent-qg7wc\" (UID: \"d21ccc94-60a4-4663-bbf1-a243a95bfa44\") " pod="kube-system/konnectivity-agent-qg7wc" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066630 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-sysctl-conf\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066709 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-ovnkube-script-lib\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066726 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-sysctl-d\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066740 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-sysctl-conf\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066747 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066778 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7nktq\" (UniqueName: \"kubernetes.io/projected/77d4a48a-9687-4338-b49d-1219168b52e2-kube-api-access-7nktq\") pod \"node-ca-q7xm7\" (UID: \"77d4a48a-9687-4338-b49d-1219168b52e2\") " pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066810 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-run-k8s-cni-cncf-io\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066833 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-socket-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066858 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-lib-modules\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066885 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-slash\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066858 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-host-run-k8s-cni-cncf-io\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.066958 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2784dfc5-43c4-434e-be4d-5698f348de7a-socket-dir\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.069076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.067022 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-lib-modules\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.069623 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.069077 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-tmp\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.069623 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.069103 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-etc-tuned\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.069623 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.069242 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d21ccc94-60a4-4663-bbf1-a243a95bfa44-agent-certs\") pod \"konnectivity-agent-qg7wc\" (UID: \"d21ccc94-60a4-4663-bbf1-a243a95bfa44\") " pod="kube-system/konnectivity-agent-qg7wc" May 26 16:00:50.080029 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.079572 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9n7h\" (UniqueName: \"kubernetes.io/projected/7b7040cd-6bfe-4c91-aa1a-59981592b6f2-kube-api-access-l9n7h\") pod \"tuned-t495h\" (UID: \"7b7040cd-6bfe-4c91-aa1a-59981592b6f2\") " pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.080029 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.079980 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6r9\" (UniqueName: \"kubernetes.io/projected/3f256df7-c5ad-4a87-8369-dace4f5d89d1-kube-api-access-sf6r9\") pod \"iptables-alerter-p72d6\" (UID: \"3f256df7-c5ad-4a87-8369-dace4f5d89d1\") " pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:50.080342 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.080299 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzs9f\" (UniqueName: \"kubernetes.io/projected/2784dfc5-43c4-434e-be4d-5698f348de7a-kube-api-access-dzs9f\") pod \"aws-ebs-csi-driver-node-b8zqp\" (UID: \"2784dfc5-43c4-434e-be4d-5698f348de7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.080682 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.080644 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wh7\" (UniqueName: \"kubernetes.io/projected/442af588-81ae-4c16-bafd-6a466acbaf11-kube-api-access-49wh7\") pod \"multus-additional-cni-plugins-z2msh\" (UID: \"442af588-81ae-4c16-bafd-6a466acbaf11\") " pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.081476 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.081453 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4fd\" (UniqueName: \"kubernetes.io/projected/ddf6bd23-ce8a-4927-b11f-c552d481bc33-kube-api-access-wb4fd\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:50.081789 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.081769 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nktq\" (UniqueName: \"kubernetes.io/projected/77d4a48a-9687-4338-b49d-1219168b52e2-kube-api-access-7nktq\") pod \"node-ca-q7xm7\" (UID: \"77d4a48a-9687-4338-b49d-1219168b52e2\") " pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:50.082009 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.081988 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxxz\" (UniqueName: \"kubernetes.io/projected/dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee-kube-api-access-9bxxz\") pod \"multus-l6rc8\" (UID: \"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee\") " pod="openshift-multus/multus-l6rc8" May 26 16:00:50.101982 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.101961 2567 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:50.167111 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167076 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-run-systemd\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167133 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-run-openvswitch\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167152 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167172 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-var-lib-openvswitch\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167188 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-run-systemd\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167194 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3874d0e8-75c4-4fed-be1a-f305a44abfa8-dbus\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167188 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-run-openvswitch\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167213 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-run-ovn-kubernetes\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167223 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167248 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-run-ovn-kubernetes\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167247 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-cni-bin\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167272 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-var-lib-openvswitch\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167297 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-ovn-node-metrics-cert\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167318 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-kubelet\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167333 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-run-netns\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167352 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-cni-netd\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167371 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-env-overrides\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167285 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-cni-bin\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167386 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3874d0e8-75c4-4fed-be1a-f305a44abfa8-dbus\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167392 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-kubelet\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167397 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r45ll\" (UniqueName: \"kubernetes.io/projected/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-kube-api-access-r45ll\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167413 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-run-netns\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167417 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-cni-netd\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167534 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-etc-openvswitch\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167589 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-systemd-units\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167613 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-run-ovn\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167648 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-etc-openvswitch\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.167678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167665 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-systemd-units\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167642 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3874d0e8-75c4-4fed-be1a-f305a44abfa8-kubelet-config\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167710 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3874d0e8-75c4-4fed-be1a-f305a44abfa8-kubelet-config\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167721 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-log-socket\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167674 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-run-ovn\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167737 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-ovnkube-config\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167758 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-ovnkube-script-lib\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167761 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-log-socket\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167784 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167827 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-slash\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167844 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-env-overrides\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167854 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-node-log\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167879 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167894 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-host-slash\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.167950 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-node-log\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.167993 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.168042 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret podName:3874d0e8-75c4-4fed-be1a-f305a44abfa8 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:50.668027095 +0000 UTC m=+3.230760020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret") pod "global-pull-secret-syncer-6nfnx" (UID: "3874d0e8-75c4-4fed-be1a-f305a44abfa8") : object "kube-system"/"original-pull-secret" not registered May 26 16:00:50.168458 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.168228 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-ovnkube-config\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.168968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.168314 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-ovnkube-script-lib\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.169994 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.169979 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-ovn-node-metrics-cert\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.174473 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.174453 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:00:50.174473 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.174473 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:00:50.174639 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.174483 2567 projected.go:194] Error preparing data for projected volume kube-api-access-nssvk for pod openshift-network-diagnostics/network-check-target-r6sp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:50.174639 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.174538 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk podName:86a247ed-5b25-4bd5-86f5-4ebca2a54f44 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:50.674525377 +0000 UTC m=+3.237258315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nssvk" (UniqueName: "kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk") pod "network-check-target-r6sp2" (UID: "86a247ed-5b25-4bd5-86f5-4ebca2a54f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:50.176178 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.176157 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r45ll\" (UniqueName: \"kubernetes.io/projected/cec49982-8c45-4e9a-a2f1-9d7b6bf64030-kube-api-access-r45ll\") pod \"ovnkube-node-vflns\" (UID: \"cec49982-8c45-4e9a-a2f1-9d7b6bf64030\") " pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.257781 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.257742 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t495h" May 26 16:00:50.264750 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.264578 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q7xm7" May 26 16:00:50.273121 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.273096 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z2msh" May 26 16:00:50.277702 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.277682 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l6rc8" May 26 16:00:50.284269 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.284252 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p72d6" May 26 16:00:50.290797 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.290779 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qg7wc" May 26 16:00:50.296371 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.296355 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" May 26 16:00:50.302047 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.302025 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:00:50.570615 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.570522 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:50.570764 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.570685 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:50.570764 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.570753 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs podName:ddf6bd23-ce8a-4927-b11f-c552d481bc33 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:51.570734306 +0000 UTC m=+4.133467241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs") pod "network-metrics-daemon-x7p82" (UID: "ddf6bd23-ce8a-4927-b11f-c552d481bc33") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:50.628416 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:50.628387 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcac5bd_82bf_42d1_a0a4_61734ba3a6ee.slice/crio-4376d690db93931a6ddc23b8ac58401bc14bff96a437c91b573a475c3df8b8b9 WatchSource:0}: Error finding container 4376d690db93931a6ddc23b8ac58401bc14bff96a437c91b573a475c3df8b8b9: Status 404 returned error can't find the container with id 4376d690db93931a6ddc23b8ac58401bc14bff96a437c91b573a475c3df8b8b9 May 26 16:00:50.630479 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:50.630450 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21ccc94_60a4_4663_bbf1_a243a95bfa44.slice/crio-87b0422bd0b6e98805674648a4a7a0a5882610158a0956caa2348a6a53153ed6 WatchSource:0}: Error finding container 87b0422bd0b6e98805674648a4a7a0a5882610158a0956caa2348a6a53153ed6: Status 404 returned error can't find the container with id 87b0422bd0b6e98805674648a4a7a0a5882610158a0956caa2348a6a53153ed6 May 26 16:00:50.630698 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:50.630671 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec49982_8c45_4e9a_a2f1_9d7b6bf64030.slice/crio-3908a7701242db5b454cd9d0008453be7aaaef2aedb7a93b02ac8ad6f299ec3f WatchSource:0}: Error finding container 3908a7701242db5b454cd9d0008453be7aaaef2aedb7a93b02ac8ad6f299ec3f: Status 404 returned error can't find the container with id 3908a7701242db5b454cd9d0008453be7aaaef2aedb7a93b02ac8ad6f299ec3f May 26 16:00:50.634020 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:50.634001 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442af588_81ae_4c16_bafd_6a466acbaf11.slice/crio-4b456493e264138c312185da77dca9583d69784a38403e745ac6e082f7e44ad6 WatchSource:0}: Error finding container 4b456493e264138c312185da77dca9583d69784a38403e745ac6e082f7e44ad6: Status 404 returned error can't find the container with id 4b456493e264138c312185da77dca9583d69784a38403e745ac6e082f7e44ad6 May 26 16:00:50.635141 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:50.634827 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2784dfc5_43c4_434e_be4d_5698f348de7a.slice/crio-f987d03eb3a41aa0f8a9ddb46fefc558885b3864899c43e9fd19e15e23de2c45 WatchSource:0}: Error finding container f987d03eb3a41aa0f8a9ddb46fefc558885b3864899c43e9fd19e15e23de2c45: Status 404 returned error can't find the container with id f987d03eb3a41aa0f8a9ddb46fefc558885b3864899c43e9fd19e15e23de2c45 May 26 16:00:50.635747 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:50.635727 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d4a48a_9687_4338_b49d_1219168b52e2.slice/crio-4df75e5591d3cacc4486d8785b9772547c9d3352213ae16e4a2c64daf9c0c8b3 WatchSource:0}: Error finding container 4df75e5591d3cacc4486d8785b9772547c9d3352213ae16e4a2c64daf9c0c8b3: Status 404 returned error can't find the container with id 4df75e5591d3cacc4486d8785b9772547c9d3352213ae16e4a2c64daf9c0c8b3 May 26 16:00:50.636573 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:50.636552 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7040cd_6bfe_4c91_aa1a_59981592b6f2.slice/crio-d227e37a15087a8386bee5dd511d980b63b260e049fbc7ca09a1a1f6c1cff778 WatchSource:0}: Error finding container d227e37a15087a8386bee5dd511d980b63b260e049fbc7ca09a1a1f6c1cff778: Status 404 returned error can't find the container with id d227e37a15087a8386bee5dd511d980b63b260e049fbc7ca09a1a1f6c1cff778 May 26 16:00:50.638385 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:00:50.638353 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f256df7_c5ad_4a87_8369_dace4f5d89d1.slice/crio-b269652ab277619746bcfb47986929f7a7cd3f2824a731b070a48f5b07bf2048 WatchSource:0}: Error finding container b269652ab277619746bcfb47986929f7a7cd3f2824a731b070a48f5b07bf2048: Status 404 returned error can't find the container with id b269652ab277619746bcfb47986929f7a7cd3f2824a731b070a48f5b07bf2048 May 26 16:00:50.671430 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.671404 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:50.671546 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.671527 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:00:50.671623 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.671612 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret podName:3874d0e8-75c4-4fed-be1a-f305a44abfa8 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:51.671592186 +0000 UTC m=+4.234325108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret") pod "global-pull-secret-syncer-6nfnx" (UID: "3874d0e8-75c4-4fed-be1a-f305a44abfa8") : object "kube-system"/"original-pull-secret" not registered May 26 16:00:50.772741 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:50.772707 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:50.772872 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.772846 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:00:50.772872 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.772870 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:00:50.772958 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.772880 2567 projected.go:194] Error preparing data for projected volume kube-api-access-nssvk for pod openshift-network-diagnostics/network-check-target-r6sp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:50.772958 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:50.772946 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk podName:86a247ed-5b25-4bd5-86f5-4ebca2a54f44 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:51.772917772 +0000 UTC m=+4.335650693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nssvk" (UniqueName: "kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk") pod "network-check-target-r6sp2" (UID: "86a247ed-5b25-4bd5-86f5-4ebca2a54f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:51.000407 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.000361 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:48 +0000 UTC" deadline="2027-11-22 17:52:48.613151564 +0000 UTC" May 26 16:00:51.000407 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.000403 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13081h51m57.612753278s" May 26 16:00:51.056211 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.056149 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p72d6" event={"ID":"3f256df7-c5ad-4a87-8369-dace4f5d89d1","Type":"ContainerStarted","Data":"b269652ab277619746bcfb47986929f7a7cd3f2824a731b070a48f5b07bf2048"} May 26 16:00:51.058982 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.058949 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t495h" event={"ID":"7b7040cd-6bfe-4c91-aa1a-59981592b6f2","Type":"ContainerStarted","Data":"d227e37a15087a8386bee5dd511d980b63b260e049fbc7ca09a1a1f6c1cff778"} May 26 16:00:51.062166 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.062135 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q7xm7" event={"ID":"77d4a48a-9687-4338-b49d-1219168b52e2","Type":"ContainerStarted","Data":"4df75e5591d3cacc4486d8785b9772547c9d3352213ae16e4a2c64daf9c0c8b3"} May 26 16:00:51.066974 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.066896 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" event={"ID":"2784dfc5-43c4-434e-be4d-5698f348de7a","Type":"ContainerStarted","Data":"f987d03eb3a41aa0f8a9ddb46fefc558885b3864899c43e9fd19e15e23de2c45"} May 26 16:00:51.070974 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.070274 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal" event={"ID":"4f187ab729b0a5371eb7618f4a50c541","Type":"ContainerStarted","Data":"790fa0b41de2c8c0223aa25faeadc7ca5f7fd2f1c9ccbad64ece2b4224da0a77"} May 26 16:00:51.077103 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.077075 2567 generic.go:358] "Generic (PLEG): container finished" podID="86e0aea22240a288454e3db6d47c9448" containerID="9b3f559cb87690f99be17bb44842f06233f0447b1139a8355c9efdd4f2976550" exitCode=0 May 26 16:00:51.077203 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.077154 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" event={"ID":"86e0aea22240a288454e3db6d47c9448","Type":"ContainerDied","Data":"9b3f559cb87690f99be17bb44842f06233f0447b1139a8355c9efdd4f2976550"} May 26 16:00:51.083699 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.083606 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-126.ec2.internal" podStartSLOduration=2.083590651 podStartE2EDuration="2.083590651s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:51.083108339 +0000 UTC m=+3.645841281" watchObservedRunningTime="2026-05-26 16:00:51.083590651 +0000 UTC m=+3.646323596" May 26 16:00:51.084629 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.084576 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerStarted","Data":"4b456493e264138c312185da77dca9583d69784a38403e745ac6e082f7e44ad6"} May 26 16:00:51.087098 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.086446 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerStarted","Data":"3908a7701242db5b454cd9d0008453be7aaaef2aedb7a93b02ac8ad6f299ec3f"} May 26 16:00:51.093848 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.093123 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qg7wc" event={"ID":"d21ccc94-60a4-4663-bbf1-a243a95bfa44","Type":"ContainerStarted","Data":"87b0422bd0b6e98805674648a4a7a0a5882610158a0956caa2348a6a53153ed6"} May 26 16:00:51.098566 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.098540 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l6rc8" event={"ID":"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee","Type":"ContainerStarted","Data":"4376d690db93931a6ddc23b8ac58401bc14bff96a437c91b573a475c3df8b8b9"} May 26 16:00:51.582712 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.582673 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:51.582949 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:51.582910 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:51.583027 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:51.582994 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs podName:ddf6bd23-ce8a-4927-b11f-c552d481bc33 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:53.582973871 +0000 UTC m=+6.145706823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs") pod "network-metrics-daemon-x7p82" (UID: "ddf6bd23-ce8a-4927-b11f-c552d481bc33") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:51.683335 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.683298 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:51.683508 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:51.683483 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:00:51.683561 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:51.683547 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret podName:3874d0e8-75c4-4fed-be1a-f305a44abfa8 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:53.683530664 +0000 UTC m=+6.246263599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret") pod "global-pull-secret-syncer-6nfnx" (UID: "3874d0e8-75c4-4fed-be1a-f305a44abfa8") : object "kube-system"/"original-pull-secret" not registered May 26 16:00:51.784681 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:51.784641 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:51.785056 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:51.784880 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:00:51.785056 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:51.784902 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:00:51.785056 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:51.784914 2567 projected.go:194] Error preparing data for projected volume kube-api-access-nssvk for pod openshift-network-diagnostics/network-check-target-r6sp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:51.785056 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:51.784989 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk podName:86a247ed-5b25-4bd5-86f5-4ebca2a54f44 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:53.784972074 +0000 UTC m=+6.347705028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nssvk" (UniqueName: "kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk") pod "network-check-target-r6sp2" (UID: "86a247ed-5b25-4bd5-86f5-4ebca2a54f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:52.045295 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:52.045256 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:52.045743 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:52.045374 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:00:52.045886 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:52.045853 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:52.045990 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:52.045969 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:00:52.046122 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:52.046109 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:52.046224 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:52.046207 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:00:52.116372 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:52.116336 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" event={"ID":"86e0aea22240a288454e3db6d47c9448","Type":"ContainerStarted","Data":"7691d44443b5826917162f7e2a7a58ac0d37d94f2f45befeb2729029ab5c10fb"} May 26 16:00:53.600294 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:53.600251 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:53.600745 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:53.600486 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:53.600745 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:53.600561 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs podName:ddf6bd23-ce8a-4927-b11f-c552d481bc33 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.600540135 +0000 UTC m=+10.163273073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs") pod "network-metrics-daemon-x7p82" (UID: "ddf6bd23-ce8a-4927-b11f-c552d481bc33") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:53.701120 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:53.701083 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:53.701300 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:53.701270 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:00:53.701367 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:53.701351 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret podName:3874d0e8-75c4-4fed-be1a-f305a44abfa8 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.701330293 +0000 UTC m=+10.264063215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret") pod "global-pull-secret-syncer-6nfnx" (UID: "3874d0e8-75c4-4fed-be1a-f305a44abfa8") : object "kube-system"/"original-pull-secret" not registered May 26 16:00:53.802191 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:53.801474 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:53.802191 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:53.801688 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:00:53.802191 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:53.801707 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:00:53.802191 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:53.801720 2567 projected.go:194] Error preparing data for projected volume kube-api-access-nssvk for pod openshift-network-diagnostics/network-check-target-r6sp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:53.802191 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:53.801776 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk podName:86a247ed-5b25-4bd5-86f5-4ebca2a54f44 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.801758827 +0000 UTC m=+10.364491756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nssvk" (UniqueName: "kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk") pod "network-check-target-r6sp2" (UID: "86a247ed-5b25-4bd5-86f5-4ebca2a54f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:54.046677 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:54.046233 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:54.046677 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:54.046365 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:00:54.046677 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:54.046502 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:54.046677 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:54.046617 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:00:54.046677 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:54.046653 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:54.047092 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:54.046737 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:00:55.704431 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.704218 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-126.ec2.internal" podStartSLOduration=6.704197581 podStartE2EDuration="6.704197581s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:52.130408372 +0000 UTC m=+4.693141316" watchObservedRunningTime="2026-05-26 16:00:55.704197581 +0000 UTC m=+8.266930525" May 26 16:00:55.704898 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.704581 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-czhdz"] May 26 16:00:55.707846 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.707824 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-czhdz" May 26 16:00:55.710658 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.710464 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hqbdt\"" May 26 16:00:55.710658 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.710617 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" May 26 16:00:55.711372 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.711317 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" May 26 16:00:55.817929 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.817846 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe-tmp-dir\") pod \"node-resolver-czhdz\" (UID: \"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe\") " pod="openshift-dns/node-resolver-czhdz" May 26 16:00:55.817929 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.817906 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe-hosts-file\") pod \"node-resolver-czhdz\" (UID: \"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe\") " pod="openshift-dns/node-resolver-czhdz" May 26 16:00:55.818178 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.818010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6lg\" (UniqueName: \"kubernetes.io/projected/3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe-kube-api-access-fz6lg\") pod \"node-resolver-czhdz\" (UID: \"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe\") " pod="openshift-dns/node-resolver-czhdz" May 26 16:00:55.918663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.918626 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe-tmp-dir\") pod \"node-resolver-czhdz\" (UID: \"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe\") " pod="openshift-dns/node-resolver-czhdz" May 26 16:00:55.918826 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.918700 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe-hosts-file\") pod \"node-resolver-czhdz\" (UID: \"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe\") " pod="openshift-dns/node-resolver-czhdz" May 26 16:00:55.918826 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.918743 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6lg\" (UniqueName: \"kubernetes.io/projected/3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe-kube-api-access-fz6lg\") pod \"node-resolver-czhdz\" (UID: \"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe\") " pod="openshift-dns/node-resolver-czhdz" May 26 16:00:55.918964 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.918873 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe-hosts-file\") pod \"node-resolver-czhdz\" (UID: \"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe\") " pod="openshift-dns/node-resolver-czhdz" May 26 16:00:55.919086 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.919061 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe-tmp-dir\") pod \"node-resolver-czhdz\" (UID: \"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe\") " pod="openshift-dns/node-resolver-czhdz" May 26 16:00:55.929442 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:55.929385 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6lg\" (UniqueName: \"kubernetes.io/projected/3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe-kube-api-access-fz6lg\") pod \"node-resolver-czhdz\" (UID: \"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe\") " pod="openshift-dns/node-resolver-czhdz" May 26 16:00:56.019504 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:56.019085 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-czhdz" May 26 16:00:56.046216 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:56.046129 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:56.046216 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:56.046154 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:56.046425 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:56.046264 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:00:56.046425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:56.046287 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:56.046425 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:56.046376 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:00:56.046581 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:56.046462 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:00:57.632525 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:57.632479 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:57.633021 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:57.632688 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:57.633021 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:57.632761 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs podName:ddf6bd23-ce8a-4927-b11f-c552d481bc33 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:05.632742384 +0000 UTC m=+18.195475324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs") pod "network-metrics-daemon-x7p82" (UID: "ddf6bd23-ce8a-4927-b11f-c552d481bc33") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:57.733915 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:57.733874 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:57.734112 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:57.734051 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:00:57.734170 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:57.734140 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret podName:3874d0e8-75c4-4fed-be1a-f305a44abfa8 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:05.734116992 +0000 UTC m=+18.296849931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret") pod "global-pull-secret-syncer-6nfnx" (UID: "3874d0e8-75c4-4fed-be1a-f305a44abfa8") : object "kube-system"/"original-pull-secret" not registered May 26 16:00:57.834483 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:57.834431 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:57.834704 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:57.834680 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:00:57.834821 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:57.834713 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:00:57.834821 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:57.834729 2567 projected.go:194] Error preparing data for projected volume kube-api-access-nssvk for pod openshift-network-diagnostics/network-check-target-r6sp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:57.834821 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:57.834802 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk podName:86a247ed-5b25-4bd5-86f5-4ebca2a54f44 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:05.834775789 +0000 UTC m=+18.397508716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nssvk" (UniqueName: "kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk") pod "network-check-target-r6sp2" (UID: "86a247ed-5b25-4bd5-86f5-4ebca2a54f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:58.047431 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:58.046846 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:00:58.047431 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:58.046987 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:00:58.047431 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:58.047066 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:00:58.047431 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:58.047183 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:00:58.047431 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:00:58.047225 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:00:58.047431 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:00:58.047290 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:00.045517 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:00.045481 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:00.045968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:00.045481 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:00.045968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:00.045482 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:00.045968 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:00.045705 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:00.045968 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:00.045812 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:00.045968 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:00.045886 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:02.046226 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:02.046177 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:02.046645 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:02.046256 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:02.046645 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:02.046360 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:02.046645 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:02.046362 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:02.046645 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:02.046485 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:02.046645 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:02.046574 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:04.045163 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:04.045131 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:04.045633 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:04.045131 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:04.045633 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:04.045250 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:04.045633 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:04.045346 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:04.045633 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:04.045134 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:04.045633 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:04.045430 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:05.697071 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:05.697024 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:05.697502 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:05.697220 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:05.697502 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:05.697291 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs podName:ddf6bd23-ce8a-4927-b11f-c552d481bc33 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:21.69727543 +0000 UTC m=+34.260008356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs") pod "network-metrics-daemon-x7p82" (UID: "ddf6bd23-ce8a-4927-b11f-c552d481bc33") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:05.797614 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:05.797580 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:05.797767 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:05.797727 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:01:05.797838 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:05.797793 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret podName:3874d0e8-75c4-4fed-be1a-f305a44abfa8 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:21.797776423 +0000 UTC m=+34.360509345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret") pod "global-pull-secret-syncer-6nfnx" (UID: "3874d0e8-75c4-4fed-be1a-f305a44abfa8") : object "kube-system"/"original-pull-secret" not registered May 26 16:01:05.898185 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:05.898151 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:05.898355 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:05.898331 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:01:05.898409 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:05.898357 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:01:05.898409 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:05.898371 2567 projected.go:194] Error preparing data for projected volume kube-api-access-nssvk for pod openshift-network-diagnostics/network-check-target-r6sp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:05.898495 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:05.898435 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk podName:86a247ed-5b25-4bd5-86f5-4ebca2a54f44 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:21.898416727 +0000 UTC m=+34.461149653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nssvk" (UniqueName: "kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk") pod "network-check-target-r6sp2" (UID: "86a247ed-5b25-4bd5-86f5-4ebca2a54f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:06.045726 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:06.045564 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:06.045726 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:06.045584 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:06.045726 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:06.045617 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:06.045982 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:06.045728 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:06.045982 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:06.045791 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:06.045982 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:06.045874 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:08.046245 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:08.046217 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:08.046618 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:08.046300 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:08.046618 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:08.046383 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:08.046618 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:08.046476 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:08.046618 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:08.046510 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:08.046618 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:08.046557 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:08.682454 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:01:08.682424 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a30554b_dd8f_4aa1_a4ec_e4bc64b466fe.slice/crio-4bf663e4923e655cdda32813f1259739842113d53ce3ab37fef8f9343e65bd2a WatchSource:0}: Error finding container 4bf663e4923e655cdda32813f1259739842113d53ce3ab37fef8f9343e65bd2a: Status 404 returned error can't find the container with id 4bf663e4923e655cdda32813f1259739842113d53ce3ab37fef8f9343e65bd2a May 26 16:01:09.142112 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.141724 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t495h" event={"ID":"7b7040cd-6bfe-4c91-aa1a-59981592b6f2","Type":"ContainerStarted","Data":"ac54711d97a30226d832f6125184313b5908338ad12ee8638bf34d350199335f"} May 26 16:01:09.142989 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.142964 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q7xm7" event={"ID":"77d4a48a-9687-4338-b49d-1219168b52e2","Type":"ContainerStarted","Data":"cb051b536a1d60d95c56338303f4acecd8b718798a22b0e9eb7c359b9b76f80c"} May 26 16:01:09.144038 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.144018 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" event={"ID":"2784dfc5-43c4-434e-be4d-5698f348de7a","Type":"ContainerStarted","Data":"82bd668c05ee22243b785678736458cb24d07f6dd630b09e06984a4495d93528"} May 26 16:01:09.145219 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.145201 2567 generic.go:358] "Generic (PLEG): container finished" podID="442af588-81ae-4c16-bafd-6a466acbaf11" containerID="21e0cb2d785c6b53c69903b529eec47953457f81e012ea8b5ad9318ebe578c87" exitCode=0 May 26 16:01:09.145297 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.145253 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerDied","Data":"21e0cb2d785c6b53c69903b529eec47953457f81e012ea8b5ad9318ebe578c87"} May 26 16:01:09.146918 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.146903 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:01:09.147253 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.147236 2567 generic.go:358] "Generic (PLEG): container finished" podID="cec49982-8c45-4e9a-a2f1-9d7b6bf64030" containerID="392f9ddc0406bd41c288246fb0f461907d4bd4b9eab31403dfadb956453979bd" exitCode=1 May 26 16:01:09.147330 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.147294 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerStarted","Data":"d930473d520075e24bdb7b92dc8def24c67f278c54823e67fbac0da6dd8e1762"} May 26 16:01:09.147330 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.147311 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerDied","Data":"392f9ddc0406bd41c288246fb0f461907d4bd4b9eab31403dfadb956453979bd"} May 26 16:01:09.147330 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.147322 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerStarted","Data":"6273038a30b0ad668b2583659c835737cb00b401c4f948f161f651b86ec9b606"} May 26 16:01:09.150375 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.150356 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qg7wc" event={"ID":"d21ccc94-60a4-4663-bbf1-a243a95bfa44","Type":"ContainerStarted","Data":"b2dd81a1573963ea9e3b27e031d07f555aefcd4cfbe62a2d4b1e1df508f4a256"} May 26 16:01:09.151541 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.151515 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l6rc8" event={"ID":"dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee","Type":"ContainerStarted","Data":"7fcb1c523c3d49185b4c2139207020af7876884801d9554e6883316bdf5780b7"} May 26 16:01:09.152647 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.152631 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-czhdz" event={"ID":"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe","Type":"ContainerStarted","Data":"6c9b7191761e7a006cdb2d6d34fd9771e005e2b815743d8e0697ac4d0514ea35"} May 26 16:01:09.152647 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.152651 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-czhdz" event={"ID":"3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe","Type":"ContainerStarted","Data":"4bf663e4923e655cdda32813f1259739842113d53ce3ab37fef8f9343e65bd2a"} May 26 16:01:09.159829 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.159796 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t495h" podStartSLOduration=3.120148273 podStartE2EDuration="21.159785498s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.638641335 +0000 UTC m=+3.201374256" lastFinishedPulling="2026-05-26 16:01:08.67827855 +0000 UTC m=+21.241011481" observedRunningTime="2026-05-26 16:01:09.159537491 +0000 UTC m=+21.722270428" watchObservedRunningTime="2026-05-26 16:01:09.159785498 +0000 UTC m=+21.722518442" May 26 16:01:09.173061 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.173027 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-czhdz" podStartSLOduration=14.173015881 podStartE2EDuration="14.173015881s" podCreationTimestamp="2026-05-26 16:00:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:01:09.172587937 +0000 UTC m=+21.735320891" watchObservedRunningTime="2026-05-26 16:01:09.173015881 +0000 UTC m=+21.735748823" May 26 16:01:09.211154 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.211111 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qg7wc" podStartSLOduration=3.3393633400000002 podStartE2EDuration="21.211099865s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.633106673 +0000 UTC m=+3.195839599" lastFinishedPulling="2026-05-26 16:01:08.504843197 +0000 UTC m=+21.067576124" observedRunningTime="2026-05-26 16:01:09.210907913 +0000 UTC m=+21.773640856" watchObservedRunningTime="2026-05-26 16:01:09.211099865 +0000 UTC m=+21.773832805" May 26 16:01:09.226411 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.226371 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q7xm7" podStartSLOduration=3.188471733 podStartE2EDuration="21.226360292s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.637677224 +0000 UTC m=+3.200410147" lastFinishedPulling="2026-05-26 16:01:08.675565778 +0000 UTC m=+21.238298706" observedRunningTime="2026-05-26 16:01:09.225872529 +0000 UTC m=+21.788605472" watchObservedRunningTime="2026-05-26 16:01:09.226360292 +0000 UTC m=+21.789093234" May 26 16:01:09.247049 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:09.247004 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-l6rc8" podStartSLOduration=3.081946262 podStartE2EDuration="21.246991324s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.633105574 +0000 UTC m=+3.195838498" lastFinishedPulling="2026-05-26 16:01:08.798150639 +0000 UTC m=+21.360883560" observedRunningTime="2026-05-26 16:01:09.246042308 +0000 UTC m=+21.808775251" watchObservedRunningTime="2026-05-26 16:01:09.246991324 +0000 UTC m=+21.809724268" May 26 16:01:10.045799 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.045772 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:10.046004 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.045820 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:10.046004 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.045822 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:10.046004 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:10.045893 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:10.046004 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:10.045982 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:10.046215 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:10.046050 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:10.156141 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.156109 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:01:10.156520 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.156429 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerStarted","Data":"29038f8926fb7fedcbddf1f7ba354384d427c4f09fabad7ce506b80ac9d03341"} May 26 16:01:10.156520 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.156465 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerStarted","Data":"7a79fc14e3233067f648c9a59084480fd4aada775d17632154329ef85855db78"} May 26 16:01:10.156520 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.156483 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerStarted","Data":"3d53c5632b6dbd1fb834ea87a3ec443acb133daaf5ee1bdbe91d8a9695d93c63"} May 26 16:01:10.157651 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.157613 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p72d6" event={"ID":"3f256df7-c5ad-4a87-8369-dace4f5d89d1","Type":"ContainerStarted","Data":"24b8ebac1aa8dd478f90f7e1a36e4f1eab47e1dc0672fe3930fe625b4e8444cf"} May 26 16:01:10.170602 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.170566 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p72d6" podStartSLOduration=4.133598527 podStartE2EDuration="22.170553235s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.640425736 +0000 UTC m=+3.203158656" lastFinishedPulling="2026-05-26 16:01:08.677380429 +0000 UTC m=+21.240113364" observedRunningTime="2026-05-26 16:01:10.170510602 +0000 UTC m=+22.733243544" watchObservedRunningTime="2026-05-26 16:01:10.170553235 +0000 UTC m=+22.733286178" May 26 16:01:10.601376 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:10.601194 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" May 26 16:01:11.031257 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:11.031137 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T16:01:10.601370749Z","UUID":"1f330a4b-4597-41fd-a0ba-7b0048562faf","Handler":null,"Name":"","Endpoint":""} May 26 16:01:11.032814 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:11.032796 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 May 26 16:01:11.032814 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:11.032818 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock May 26 16:01:11.161272 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:11.161235 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" event={"ID":"2784dfc5-43c4-434e-be4d-5698f348de7a","Type":"ContainerStarted","Data":"d12a664697ace2c1090be8339ef6f91c9386b6bc1896363239f0d8258ac3dd10"} May 26 16:01:12.045890 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.045819 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:12.046174 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.045814 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:12.046174 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:12.045949 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:12.046174 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:12.046013 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:12.046174 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.045814 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:12.046174 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:12.046084 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:12.131714 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.131669 2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qg7wc" May 26 16:01:12.132312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.132287 2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qg7wc" May 26 16:01:12.165892 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.165864 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:01:12.166553 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.166287 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerStarted","Data":"93b554974060dda9e0149e3a0e6756da697ebee908a219fa0407e044808a784a"} May 26 16:01:12.168167 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.168138 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" event={"ID":"2784dfc5-43c4-434e-be4d-5698f348de7a","Type":"ContainerStarted","Data":"19832ce1378c56609352370538c6476821aac5d7abd037a3a954caecedf19649"} May 26 16:01:12.168738 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.168381 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qg7wc" May 26 16:01:12.168960 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.168923 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qg7wc" May 26 16:01:12.202659 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:12.202611 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8zqp" podStartSLOduration=3.123567544 podStartE2EDuration="24.202598909s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.636686003 +0000 UTC m=+3.199418942" lastFinishedPulling="2026-05-26 16:01:11.715717386 +0000 UTC m=+24.278450307" observedRunningTime="2026-05-26 16:01:12.188541841 +0000 UTC m=+24.751274784" watchObservedRunningTime="2026-05-26 16:01:12.202598909 +0000 UTC m=+24.765331864" May 26 16:01:14.045925 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:14.045889 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:14.045925 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:14.045914 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:14.046538 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:14.045889 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:14.046538 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:14.046036 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:14.046538 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:14.046198 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:14.046538 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:14.046289 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:15.174751 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:15.174571 2567 generic.go:358] "Generic (PLEG): container finished" podID="442af588-81ae-4c16-bafd-6a466acbaf11" containerID="5d42f5e2d65e60dc759b08d33ca83381c64ceb3e69170ad47163a90fad70d9d3" exitCode=0 May 26 16:01:15.175189 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:15.174657 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerDied","Data":"5d42f5e2d65e60dc759b08d33ca83381c64ceb3e69170ad47163a90fad70d9d3"} May 26 16:01:15.177837 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:15.177822 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:01:15.178205 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:15.178177 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerStarted","Data":"aa4c265716b55563166250fd51645497ee13b92fc1d03eaa2587d63f717fe532"} May 26 16:01:15.178464 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:15.178444 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:01:15.178533 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:15.178471 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:01:15.178607 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:15.178590 2567 scope.go:117] "RemoveContainer" containerID="392f9ddc0406bd41c288246fb0f461907d4bd4b9eab31403dfadb956453979bd" May 26 16:01:15.194200 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:15.194182 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:01:16.045834 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.045803 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:16.046024 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.045803 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:16.046024 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:16.045917 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:16.046133 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:16.046024 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:16.046133 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.045804 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:16.046219 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:16.046152 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:16.183183 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.183158 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:01:16.183524 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.183439 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" event={"ID":"cec49982-8c45-4e9a-a2f1-9d7b6bf64030","Type":"ContainerStarted","Data":"fb3fe33150709d0b6e6e8483733981dbd4f32bb3902d1ca0a2c84e4da0c66860"} May 26 16:01:16.183771 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.183753 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:01:16.198411 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.198387 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:01:16.212329 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.212286 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" podStartSLOduration=10.12452756 podStartE2EDuration="28.212273893s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.633315955 +0000 UTC m=+3.196048892" lastFinishedPulling="2026-05-26 16:01:08.721062294 +0000 UTC m=+21.283795225" observedRunningTime="2026-05-26 16:01:16.211978308 +0000 UTC m=+28.774711252" watchObservedRunningTime="2026-05-26 16:01:16.212273893 +0000 UTC m=+28.775006837" May 26 16:01:16.382879 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.382637 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6nfnx"] May 26 16:01:16.383059 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.383007 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:16.383137 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:16.383114 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:16.384825 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.384795 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7p82"] May 26 16:01:16.384959 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.384837 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r6sp2"] May 26 16:01:16.384959 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.384922 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:16.385070 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:16.385027 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:16.385128 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:16.385119 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:16.385223 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:16.385201 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:17.187142 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:17.187107 2567 generic.go:358] "Generic (PLEG): container finished" podID="442af588-81ae-4c16-bafd-6a466acbaf11" containerID="75339013d7d9f898f28d468529f16bfe5cfc863e73eea2ccbb07078b0a6fd0f5" exitCode=0 May 26 16:01:17.187696 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:17.187191 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerDied","Data":"75339013d7d9f898f28d468529f16bfe5cfc863e73eea2ccbb07078b0a6fd0f5"} May 26 16:01:18.047409 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:18.047317 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:18.047535 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:18.047435 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:18.047535 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:18.047509 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:18.047687 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:18.047654 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:18.047751 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:18.047682 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:18.047866 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:18.047842 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:18.191269 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:18.191216 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerStarted","Data":"267cd8a3f66732b2b0a9c72ab20279b516de66caf306cdf1e31df70819bf77ad"} May 26 16:01:19.194875 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:19.194837 2567 generic.go:358] "Generic (PLEG): container finished" podID="442af588-81ae-4c16-bafd-6a466acbaf11" containerID="267cd8a3f66732b2b0a9c72ab20279b516de66caf306cdf1e31df70819bf77ad" exitCode=0 May 26 16:01:19.195374 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:19.194903 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerDied","Data":"267cd8a3f66732b2b0a9c72ab20279b516de66caf306cdf1e31df70819bf77ad"} May 26 16:01:20.045581 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:20.045546 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:20.045727 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:20.045652 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nfnx" podUID="3874d0e8-75c4-4fed-be1a-f305a44abfa8" May 26 16:01:20.045727 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:20.045663 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:20.045727 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:20.045688 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:20.045897 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:20.045766 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6sp2" podUID="86a247ed-5b25-4bd5-86f5-4ebca2a54f44" May 26 16:01:20.045897 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:20.045860 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:01:21.713269 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.713074 2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-126.ec2.internal" event="NodeReady" May 26 16:01:21.713727 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.713391 2567 kubelet_node_status.go:550] "Fast updating node status as it just became ready" May 26 16:01:21.723634 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.723599 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:21.723788 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:21.723750 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:21.723838 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:21.723809 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs podName:ddf6bd23-ce8a-4927-b11f-c552d481bc33 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.723793099 +0000 UTC m=+66.286526025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs") pod "network-metrics-daemon-x7p82" (UID: "ddf6bd23-ce8a-4927-b11f-c552d481bc33") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:21.745319 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.745290 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz"] May 26 16:01:21.763261 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.763221 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd"] May 26 16:01:21.764599 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.763464 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:21.766621 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.766436 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" May 26 16:01:21.766621 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.766466 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tszrn\"" May 26 16:01:21.766621 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.766466 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" May 26 16:01:21.778512 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.778480 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-585cd67447-r2kt6"] May 26 16:01:21.779088 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.779069 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" May 26 16:01:21.781320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.781297 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zhwsn\"" May 26 16:01:21.781422 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.781345 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" May 26 16:01:21.781422 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.781389 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" May 26 16:01:21.781709 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.781689 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" May 26 16:01:21.781822 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.781762 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" May 26 16:01:21.791810 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.791789 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn"] May 26 16:01:21.791967 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.791928 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:21.794195 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.794171 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" May 26 16:01:21.794195 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.794195 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" May 26 16:01:21.794343 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.794171 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" May 26 16:01:21.794343 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.794172 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nv2cx\"" May 26 16:01:21.804702 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.804674 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" May 26 16:01:21.805338 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.805317 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq"] May 26 16:01:21.805484 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.805467 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:21.807504 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.807486 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" May 26 16:01:21.807999 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.807870 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" May 26 16:01:21.807999 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.807888 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" May 26 16:01:21.807999 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.807984 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" May 26 16:01:21.824925 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.824895 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:21.825076 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:21.825022 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:01:21.825130 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:21.825091 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret podName:3874d0e8-75c4-4fed-be1a-f305a44abfa8 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.825070275 +0000 UTC m=+66.387803200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret") pod "global-pull-secret-syncer-6nfnx" (UID: "3874d0e8-75c4-4fed-be1a-f305a44abfa8") : object "kube-system"/"original-pull-secret" not registered May 26 16:01:21.825695 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.825671 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2kl9l"] May 26 16:01:21.825847 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.825829 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:21.827815 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.827798 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" May 26 16:01:21.844605 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.844571 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz"] May 26 16:01:21.844747 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.844634 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2kl9l" May 26 16:01:21.844747 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.844679 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd"] May 26 16:01:21.844747 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.844708 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq"] May 26 16:01:21.844876 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.844840 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn"] May 26 16:01:21.844876 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.844864 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585cd67447-r2kt6"] May 26 16:01:21.844987 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.844879 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rl5nn"] May 26 16:01:21.846846 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.846824 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lppkw\"" May 26 16:01:21.846846 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.846843 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" May 26 16:01:21.847033 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.846843 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" May 26 16:01:21.866160 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.866135 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2kl9l"] May 26 16:01:21.866160 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.866164 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rl5nn"] May 26 16:01:21.866333 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.866280 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:21.869273 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.869252 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" May 26 16:01:21.869386 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.869348 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" May 26 16:01:21.869386 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.869257 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" May 26 16:01:21.869498 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.869257 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-22s8h\"" May 26 16:01:21.925385 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:21.925385 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925388 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-bound-sa-token\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:21.925579 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925418 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-kube-api-access-p4vdw\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:21.925579 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8aee9c3-623c-4438-934c-ea0e437eda7a-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:21.925579 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925516 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e7021e8-c7c5-426d-8214-1a923d10e9fd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c9b665b78-6h4qd\" (UID: \"0e7021e8-c7c5-426d-8214-1a923d10e9fd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" May 26 16:01:21.925579 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925545 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-hub\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:21.925579 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925573 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb9bh\" (UniqueName: \"kubernetes.io/projected/666c741a-5087-4f81-aee0-991abc9556bc-kube-api-access-vb9bh\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:21.925820 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925599 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbqc\" (UniqueName: \"kubernetes.io/projected/70fb5ba8-e654-4f66-8a9a-597dec2fde25-kube-api-access-mnbqc\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:21.925820 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925653 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:21.925820 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925675 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-certificates\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:21.925820 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925713 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35b7a055-0f8a-49b5-86f8-dd00b288fb0b-tmp\") pod \"klusterlet-addon-workmgr-7b955d89fd-cw5tq\" (UID: \"35b7a055-0f8a-49b5-86f8-dd00b288fb0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:21.925820 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925757 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-ca-trust-extracted\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:21.925820 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925784 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-ca\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:21.925820 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925814 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925834 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:21.925838 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:21.925864 2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925864 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-installation-pull-secrets\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:21.925877 2567 projected.go:194] Error preparing data for projected volume kube-api-access-nssvk for pod openshift-network-diagnostics/network-check-target-r6sp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925889 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925906 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz9mw\" (UniqueName: \"kubernetes.io/projected/0e7021e8-c7c5-426d-8214-1a923d10e9fd-kube-api-access-mz9mw\") pod \"managed-serviceaccount-addon-agent-5c9b665b78-6h4qd\" (UID: \"0e7021e8-c7c5-426d-8214-1a923d10e9fd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.925930 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70fb5ba8-e654-4f66-8a9a-597dec2fde25-config-volume\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:21.925968 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk podName:86a247ed-5b25-4bd5-86f5-4ebca2a54f44 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.925928225 +0000 UTC m=+66.488661151 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nssvk" (UniqueName: "kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk") pod "network-check-target-r6sp2" (UID: "86a247ed-5b25-4bd5-86f5-4ebca2a54f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.926057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/35b7a055-0f8a-49b5-86f8-dd00b288fb0b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b955d89fd-cw5tq\" (UID: \"35b7a055-0f8a-49b5-86f8-dd00b288fb0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.926081 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.926101 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhwg\" (UniqueName: \"kubernetes.io/projected/35b7a055-0f8a-49b5-86f8-dd00b288fb0b-kube-api-access-qbhwg\") pod \"klusterlet-addon-workmgr-7b955d89fd-cw5tq\" (UID: \"35b7a055-0f8a-49b5-86f8-dd00b288fb0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.926116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-trusted-ca\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:21.926135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.926140 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fb5ba8-e654-4f66-8a9a-597dec2fde25-tmp-dir\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:21.926663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.926165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-image-registry-private-configuration\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:21.926663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:21.926186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/666c741a-5087-4f81-aee0-991abc9556bc-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.026954 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.026858 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-trusted-ca\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.026954 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.026905 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fb5ba8-e654-4f66-8a9a-597dec2fde25-tmp-dir\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.026955 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.026980 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-bound-sa-token\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027007 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-kube-api-access-p4vdw\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027035 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-hub\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027062 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e7021e8-c7c5-426d-8214-1a923d10e9fd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c9b665b78-6h4qd\" (UID: \"0e7021e8-c7c5-426d-8214-1a923d10e9fd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027097 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-certificates\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027127 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35b7a055-0f8a-49b5-86f8-dd00b288fb0b-tmp\") pod \"klusterlet-addon-workmgr-7b955d89fd-cw5tq\" (UID: \"35b7a055-0f8a-49b5-86f8-dd00b288fb0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027153 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-ca-trust-extracted\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027180 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-ca\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.027208 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027207 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027235 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4m55\" (UniqueName: \"kubernetes.io/projected/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-kube-api-access-w4m55\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027254 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-installation-pull-secrets\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027272 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027298 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhwg\" (UniqueName: \"kubernetes.io/projected/35b7a055-0f8a-49b5-86f8-dd00b288fb0b-kube-api-access-qbhwg\") pod \"klusterlet-addon-workmgr-7b955d89fd-cw5tq\" (UID: \"35b7a055-0f8a-49b5-86f8-dd00b288fb0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027298 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fb5ba8-e654-4f66-8a9a-597dec2fde25-tmp-dir\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027319 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-image-registry-private-configuration\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027348 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/666c741a-5087-4f81-aee0-991abc9556bc-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027392 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbqc\" (UniqueName: \"kubernetes.io/projected/70fb5ba8-e654-4f66-8a9a-597dec2fde25-kube-api-access-mnbqc\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027416 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8aee9c3-623c-4438-934c-ea0e437eda7a-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027442 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vb9bh\" (UniqueName: \"kubernetes.io/projected/666c741a-5087-4f81-aee0-991abc9556bc-kube-api-access-vb9bh\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027507 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027539 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70fb5ba8-e654-4f66-8a9a-597dec2fde25-config-volume\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027559 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9mw\" (UniqueName: \"kubernetes.io/projected/0e7021e8-c7c5-426d-8214-1a923d10e9fd-kube-api-access-mz9mw\") pod \"managed-serviceaccount-addon-agent-5c9b665b78-6h4qd\" (UID: \"0e7021e8-c7c5-426d-8214-1a923d10e9fd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027577 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/35b7a055-0f8a-49b5-86f8-dd00b288fb0b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b955d89fd-cw5tq\" (UID: \"35b7a055-0f8a-49b5-86f8-dd00b288fb0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:22.027705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027595 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:22.028469 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.027972 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-trusted-ca\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.028469 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.028137 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35b7a055-0f8a-49b5-86f8-dd00b288fb0b-tmp\") pod \"klusterlet-addon-workmgr-7b955d89fd-cw5tq\" (UID: \"35b7a055-0f8a-49b5-86f8-dd00b288fb0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:22.028634 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.028610 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-certificates\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.028882 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70fb5ba8-e654-4f66-8a9a-597dec2fde25-config-volume\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.027127 2567 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.028996 2567 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585cd67447-r2kt6: secret "image-registry-tls" not found May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.029052 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls podName:712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:22.529032425 +0000 UTC m=+35.091765352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls") pod "image-registry-585cd67447-r2kt6" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd") : secret "image-registry-tls" not found May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.027673 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.029227 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls podName:70fb5ba8-e654-4f66-8a9a-597dec2fde25 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:22.529214104 +0000 UTC m=+35.091947025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls") pod "dns-default-2kl9l" (UID: "70fb5ba8-e654-4f66-8a9a-597dec2fde25") : secret "dns-default-metrics-tls" not found May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.029507 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-ca-trust-extracted\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.029628 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8aee9c3-623c-4438-934c-ea0e437eda7a-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.029828 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.029887 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert podName:b8aee9c3-623c-4438-934c-ea0e437eda7a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:22.529871968 +0000 UTC m=+35.092604908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-dwthz" (UID: "b8aee9c3-623c-4438-934c-ea0e437eda7a") : secret "networking-console-plugin-cert" not found May 26 16:01:22.030393 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.030342 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/666c741a-5087-4f81-aee0-991abc9556bc-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.032740 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.032712 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-image-registry-private-configuration\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.032893 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.032851 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-installation-pull-secrets\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.032893 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.032851 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/35b7a055-0f8a-49b5-86f8-dd00b288fb0b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b955d89fd-cw5tq\" (UID: \"35b7a055-0f8a-49b5-86f8-dd00b288fb0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:22.033089 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.032907 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e7021e8-c7c5-426d-8214-1a923d10e9fd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c9b665b78-6h4qd\" (UID: \"0e7021e8-c7c5-426d-8214-1a923d10e9fd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" May 26 16:01:22.033089 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.032960 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-ca\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.033721 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.033581 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-hub\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.033721 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.033684 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.034189 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.034151 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/666c741a-5087-4f81-aee0-991abc9556bc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.037424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.037152 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbqc\" (UniqueName: \"kubernetes.io/projected/70fb5ba8-e654-4f66-8a9a-597dec2fde25-kube-api-access-mnbqc\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:22.037424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.037367 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-kube-api-access-p4vdw\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.037760 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.037722 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-bound-sa-token\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.038244 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.038218 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9mw\" (UniqueName: \"kubernetes.io/projected/0e7021e8-c7c5-426d-8214-1a923d10e9fd-kube-api-access-mz9mw\") pod \"managed-serviceaccount-addon-agent-5c9b665b78-6h4qd\" (UID: \"0e7021e8-c7c5-426d-8214-1a923d10e9fd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" May 26 16:01:22.038571 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.038548 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhwg\" (UniqueName: \"kubernetes.io/projected/35b7a055-0f8a-49b5-86f8-dd00b288fb0b-kube-api-access-qbhwg\") pod \"klusterlet-addon-workmgr-7b955d89fd-cw5tq\" (UID: \"35b7a055-0f8a-49b5-86f8-dd00b288fb0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:22.038571 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.038557 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb9bh\" (UniqueName: \"kubernetes.io/projected/666c741a-5087-4f81-aee0-991abc9556bc-kube-api-access-vb9bh\") pod \"cluster-proxy-proxy-agent-56db54c59-8djvn\" (UID: \"666c741a-5087-4f81-aee0-991abc9556bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.045594 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.045573 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:22.045734 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.045710 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:22.045791 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.045779 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:22.048441 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.048407 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8ddfl\"" May 26 16:01:22.048570 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.048551 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45wxn\"" May 26 16:01:22.048661 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.048574 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 26 16:01:22.048661 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.048561 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 26 16:01:22.048848 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.048831 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 26 16:01:22.049030 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.049004 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 26 16:01:22.098772 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.098747 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" May 26 16:01:22.116246 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.116224 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:01:22.128630 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.128420 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w4m55\" (UniqueName: \"kubernetes.io/projected/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-kube-api-access-w4m55\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:22.128630 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.128529 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:22.128802 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.128704 2567 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:22.128802 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.128784 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert podName:86cf44b2-9266-4e01-a88a-b1552e2ae1bd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:22.628763584 +0000 UTC m=+35.191496522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert") pod "ingress-canary-rl5nn" (UID: "86cf44b2-9266-4e01-a88a-b1552e2ae1bd") : secret "canary-serving-cert" not found May 26 16:01:22.141969 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.141916 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4m55\" (UniqueName: \"kubernetes.io/projected/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-kube-api-access-w4m55\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:22.155775 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.155733 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:22.264647 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.264608 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd"] May 26 16:01:22.268399 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:01:22.268367 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e7021e8_c7c5_426d_8214_1a923d10e9fd.slice/crio-dd0b1cff3030171941a4dd27fc80119342c9627596cc741bd8919fac47722757 WatchSource:0}: Error finding container dd0b1cff3030171941a4dd27fc80119342c9627596cc741bd8919fac47722757: Status 404 returned error can't find the container with id dd0b1cff3030171941a4dd27fc80119342c9627596cc741bd8919fac47722757 May 26 16:01:22.277340 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.277206 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn"] May 26 16:01:22.336250 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.336220 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq"] May 26 16:01:22.339471 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:01:22.339445 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b7a055_0f8a_49b5_86f8_dd00b288fb0b.slice/crio-64d6d900d996f9fb10f17a8cdeade6f97ee0093e271fa8e127c9e05de7aaea54 WatchSource:0}: Error finding container 64d6d900d996f9fb10f17a8cdeade6f97ee0093e271fa8e127c9e05de7aaea54: Status 404 returned error can't find the container with id 64d6d900d996f9fb10f17a8cdeade6f97ee0093e271fa8e127c9e05de7aaea54 May 26 16:01:22.532229 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.532147 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:22.532229 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.532219 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:22.532439 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.532330 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:22.532439 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.532331 2567 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:01:22.532439 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.532384 2567 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585cd67447-r2kt6: secret "image-registry-tls" not found May 26 16:01:22.532439 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.532388 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found May 26 16:01:22.532439 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.532394 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:22.532439 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.532443 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls podName:712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:23.532424942 +0000 UTC m=+36.095157868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls") pod "image-registry-585cd67447-r2kt6" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd") : secret "image-registry-tls" not found May 26 16:01:22.532709 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.532465 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls podName:70fb5ba8-e654-4f66-8a9a-597dec2fde25 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:23.532450588 +0000 UTC m=+36.095183531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls") pod "dns-default-2kl9l" (UID: "70fb5ba8-e654-4f66-8a9a-597dec2fde25") : secret "dns-default-metrics-tls" not found May 26 16:01:22.532709 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.532482 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert podName:b8aee9c3-623c-4438-934c-ea0e437eda7a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:23.532473881 +0000 UTC m=+36.095206802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-dwthz" (UID: "b8aee9c3-623c-4438-934c-ea0e437eda7a") : secret "networking-console-plugin-cert" not found May 26 16:01:22.633064 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:22.633025 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:22.633218 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.633180 2567 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:22.633262 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:22.633254 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert podName:86cf44b2-9266-4e01-a88a-b1552e2ae1bd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:23.633236598 +0000 UTC m=+36.195969532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert") pod "ingress-canary-rl5nn" (UID: "86cf44b2-9266-4e01-a88a-b1552e2ae1bd") : secret "canary-serving-cert" not found May 26 16:01:23.205090 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:23.205048 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" event={"ID":"35b7a055-0f8a-49b5-86f8-dd00b288fb0b","Type":"ContainerStarted","Data":"64d6d900d996f9fb10f17a8cdeade6f97ee0093e271fa8e127c9e05de7aaea54"} May 26 16:01:23.206132 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:23.206104 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" event={"ID":"666c741a-5087-4f81-aee0-991abc9556bc","Type":"ContainerStarted","Data":"26b2cd0fdea770dd9ff2f67af054fc7280b6fffd6b26faf5a025ce929ed348e4"} May 26 16:01:23.207179 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:23.207150 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" event={"ID":"0e7021e8-c7c5-426d-8214-1a923d10e9fd","Type":"ContainerStarted","Data":"dd0b1cff3030171941a4dd27fc80119342c9627596cc741bd8919fac47722757"} May 26 16:01:23.539097 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:23.539005 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:23.539263 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:23.539131 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:23.539263 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:23.539155 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found May 26 16:01:23.539263 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:23.539165 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:23.539263 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:23.539232 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert podName:b8aee9c3-623c-4438-934c-ea0e437eda7a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.539210495 +0000 UTC m=+38.101943458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-dwthz" (UID: "b8aee9c3-623c-4438-934c-ea0e437eda7a") : secret "networking-console-plugin-cert" not found May 26 16:01:23.539263 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:23.539243 2567 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:01:23.539263 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:23.539256 2567 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585cd67447-r2kt6: secret "image-registry-tls" not found May 26 16:01:23.539511 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:23.539275 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:23.539511 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:23.539308 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls podName:712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.539291433 +0000 UTC m=+38.102024374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls") pod "image-registry-585cd67447-r2kt6" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd") : secret "image-registry-tls" not found May 26 16:01:23.539511 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:23.539326 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls podName:70fb5ba8-e654-4f66-8a9a-597dec2fde25 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.539317034 +0000 UTC m=+38.102049958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls") pod "dns-default-2kl9l" (UID: "70fb5ba8-e654-4f66-8a9a-597dec2fde25") : secret "dns-default-metrics-tls" not found May 26 16:01:23.640529 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:23.640493 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:23.640695 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:23.640672 2567 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:23.640773 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:23.640760 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert podName:86cf44b2-9266-4e01-a88a-b1552e2ae1bd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.640740691 +0000 UTC m=+38.203473612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert") pod "ingress-canary-rl5nn" (UID: "86cf44b2-9266-4e01-a88a-b1552e2ae1bd") : secret "canary-serving-cert" not found May 26 16:01:25.218817 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:25.218731 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerStarted","Data":"9014dfe182d4135f12c1deaa8f5e578053c3c909dd25fca03a3867e22ea81d98"} May 26 16:01:25.560560 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:25.560510 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:25.560715 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:25.560573 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:25.560715 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:25.560630 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:25.560821 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:25.560766 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found May 26 16:01:25.560883 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:25.560831 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert podName:b8aee9c3-623c-4438-934c-ea0e437eda7a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.560810926 +0000 UTC m=+42.123543859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-dwthz" (UID: "b8aee9c3-623c-4438-934c-ea0e437eda7a") : secret "networking-console-plugin-cert" not found May 26 16:01:25.569067 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:25.563736 2567 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:01:25.569067 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:25.563775 2567 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585cd67447-r2kt6: secret "image-registry-tls" not found May 26 16:01:25.569067 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:25.563793 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:25.569067 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:25.563849 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls podName:712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.563830686 +0000 UTC m=+42.126563609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls") pod "image-registry-585cd67447-r2kt6" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd") : secret "image-registry-tls" not found May 26 16:01:25.569067 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:25.563881 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls podName:70fb5ba8-e654-4f66-8a9a-597dec2fde25 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.563859469 +0000 UTC m=+42.126592390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls") pod "dns-default-2kl9l" (UID: "70fb5ba8-e654-4f66-8a9a-597dec2fde25") : secret "dns-default-metrics-tls" not found May 26 16:01:25.661838 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:25.661102 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:25.661838 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:25.661383 2567 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:25.661838 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:25.661454 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert podName:86cf44b2-9266-4e01-a88a-b1552e2ae1bd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.661431545 +0000 UTC m=+42.224164482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert") pod "ingress-canary-rl5nn" (UID: "86cf44b2-9266-4e01-a88a-b1552e2ae1bd") : secret "canary-serving-cert" not found May 26 16:01:26.226831 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:26.226719 2567 generic.go:358] "Generic (PLEG): container finished" podID="442af588-81ae-4c16-bafd-6a466acbaf11" containerID="9014dfe182d4135f12c1deaa8f5e578053c3c909dd25fca03a3867e22ea81d98" exitCode=0 May 26 16:01:26.226831 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:26.226791 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerDied","Data":"9014dfe182d4135f12c1deaa8f5e578053c3c909dd25fca03a3867e22ea81d98"} May 26 16:01:27.233056 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:27.233018 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerStarted","Data":"445f2dcf32a836a7da13f904426494a3b942b16c9a9fcea89cf06c38b8fdad51"} May 26 16:01:28.237338 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:28.237300 2567 generic.go:358] "Generic (PLEG): container finished" podID="442af588-81ae-4c16-bafd-6a466acbaf11" containerID="445f2dcf32a836a7da13f904426494a3b942b16c9a9fcea89cf06c38b8fdad51" exitCode=0 May 26 16:01:28.237338 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:28.237335 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerDied","Data":"445f2dcf32a836a7da13f904426494a3b942b16c9a9fcea89cf06c38b8fdad51"} May 26 16:01:29.597669 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:29.597626 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:29.598082 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:29.597804 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found May 26 16:01:29.598082 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:29.597842 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:29.598082 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:29.597881 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert podName:b8aee9c3-623c-4438-934c-ea0e437eda7a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:37.597859997 +0000 UTC m=+50.160592923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-dwthz" (UID: "b8aee9c3-623c-4438-934c-ea0e437eda7a") : secret "networking-console-plugin-cert" not found May 26 16:01:29.598082 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:29.597927 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:29.598082 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:29.597928 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:29.598082 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:29.597995 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls podName:70fb5ba8-e654-4f66-8a9a-597dec2fde25 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:37.597982959 +0000 UTC m=+50.160715887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls") pod "dns-default-2kl9l" (UID: "70fb5ba8-e654-4f66-8a9a-597dec2fde25") : secret "dns-default-metrics-tls" not found May 26 16:01:29.598082 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:29.598011 2567 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:01:29.598082 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:29.598027 2567 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585cd67447-r2kt6: secret "image-registry-tls" not found May 26 16:01:29.598082 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:29.598077 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls podName:712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:37.598060693 +0000 UTC m=+50.160793627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls") pod "image-registry-585cd67447-r2kt6" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd") : secret "image-registry-tls" not found May 26 16:01:29.698488 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:29.698454 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:29.698669 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:29.698585 2567 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:29.698669 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:29.698662 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert podName:86cf44b2-9266-4e01-a88a-b1552e2ae1bd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:37.698641594 +0000 UTC m=+50.261374520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert") pod "ingress-canary-rl5nn" (UID: "86cf44b2-9266-4e01-a88a-b1552e2ae1bd") : secret "canary-serving-cert" not found May 26 16:01:33.247253 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:33.247216 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" event={"ID":"35b7a055-0f8a-49b5-86f8-dd00b288fb0b","Type":"ContainerStarted","Data":"46b2b0a46b6421ddf1319ff5143e4d949951b88be4bb3203f7596012917fafa4"} May 26 16:01:33.247908 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:33.247424 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:33.248651 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:33.248624 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" event={"ID":"666c741a-5087-4f81-aee0-991abc9556bc","Type":"ContainerStarted","Data":"6c4650b0e74d593bd1efa521dc980b31fb66955d8aacb24c9dde3f5a5a9c2714"} May 26 16:01:33.249291 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:33.249267 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:01:33.249984 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:33.249959 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" event={"ID":"0e7021e8-c7c5-426d-8214-1a923d10e9fd","Type":"ContainerStarted","Data":"6acfc8f70af508bf96a823ca6b1cbb92f7381881d4aae8b966a8556064499977"} May 26 16:01:33.252707 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:33.252689 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2msh" event={"ID":"442af588-81ae-4c16-bafd-6a466acbaf11","Type":"ContainerStarted","Data":"cf0d6560d1b602c711d18eaa3ee2acfc94ad9a0a1776931d3aa4aadc97cc3003"} May 26 16:01:33.265968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:33.265918 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" podStartSLOduration=34.362765128 podStartE2EDuration="44.265906795s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:01:22.34161641 +0000 UTC m=+34.904349332" lastFinishedPulling="2026-05-26 16:01:32.244758075 +0000 UTC m=+44.807490999" observedRunningTime="2026-05-26 16:01:33.265776885 +0000 UTC m=+45.828509818" watchObservedRunningTime="2026-05-26 16:01:33.265906795 +0000 UTC m=+45.828639744" May 26 16:01:33.290047 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:33.290008 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z2msh" podStartSLOduration=10.881914151 podStartE2EDuration="45.289996755s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.635635631 +0000 UTC m=+3.198368556" lastFinishedPulling="2026-05-26 16:01:25.04371824 +0000 UTC m=+37.606451160" observedRunningTime="2026-05-26 16:01:33.288804982 +0000 UTC m=+45.851538068" watchObservedRunningTime="2026-05-26 16:01:33.289996755 +0000 UTC m=+45.852729697" May 26 16:01:33.316899 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:33.316862 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" podStartSLOduration=34.359741672 podStartE2EDuration="44.316850737s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:01:22.271600298 +0000 UTC m=+34.834333234" lastFinishedPulling="2026-05-26 16:01:32.228709367 +0000 UTC m=+44.791442299" observedRunningTime="2026-05-26 16:01:33.316259452 +0000 UTC m=+45.878992394" watchObservedRunningTime="2026-05-26 16:01:33.316850737 +0000 UTC m=+45.879583680" May 26 16:01:36.262670 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:36.262633 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" event={"ID":"666c741a-5087-4f81-aee0-991abc9556bc","Type":"ContainerStarted","Data":"2e7d87b38e7800e69d200734cf9a1b3402e7b067c6cef2a6938a6592c1961c46"} May 26 16:01:36.262670 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:36.262672 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" event={"ID":"666c741a-5087-4f81-aee0-991abc9556bc","Type":"ContainerStarted","Data":"b65d395359edbc1705dceca3c93b51c2318901438a18cc1282653dabd26507b4"} May 26 16:01:36.281680 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:36.281631 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" podStartSLOduration=34.167260514 podStartE2EDuration="47.281617838s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:01:22.300638013 +0000 UTC m=+34.863370947" lastFinishedPulling="2026-05-26 16:01:35.41499535 +0000 UTC m=+47.977728271" observedRunningTime="2026-05-26 16:01:36.280095718 +0000 UTC m=+48.842828684" watchObservedRunningTime="2026-05-26 16:01:36.281617838 +0000 UTC m=+48.844350807" May 26 16:01:37.665271 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:37.665235 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:37.665271 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:37.665278 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:37.665696 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:37.665316 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:37.665696 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:37.665393 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:37.665696 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:37.665461 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls podName:70fb5ba8-e654-4f66-8a9a-597dec2fde25 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.665446219 +0000 UTC m=+66.228179166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls") pod "dns-default-2kl9l" (UID: "70fb5ba8-e654-4f66-8a9a-597dec2fde25") : secret "dns-default-metrics-tls" not found May 26 16:01:37.665696 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:37.665396 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found May 26 16:01:37.665696 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:37.665539 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert podName:b8aee9c3-623c-4438-934c-ea0e437eda7a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.665527263 +0000 UTC m=+66.228260188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-dwthz" (UID: "b8aee9c3-623c-4438-934c-ea0e437eda7a") : secret "networking-console-plugin-cert" not found May 26 16:01:37.665696 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:37.665399 2567 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:01:37.665696 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:37.665557 2567 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585cd67447-r2kt6: secret "image-registry-tls" not found May 26 16:01:37.665696 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:37.665593 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls podName:712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.665583071 +0000 UTC m=+66.228316004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls") pod "image-registry-585cd67447-r2kt6" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd") : secret "image-registry-tls" not found May 26 16:01:37.766333 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:37.766294 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:37.766523 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:37.766461 2567 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:37.766587 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:37.766543 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert podName:86cf44b2-9266-4e01-a88a-b1552e2ae1bd nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.766518565 +0000 UTC m=+66.329251501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert") pod "ingress-canary-rl5nn" (UID: "86cf44b2-9266-4e01-a88a-b1552e2ae1bd") : secret "canary-serving-cert" not found May 26 16:01:48.206330 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:48.206301 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vflns" May 26 16:01:53.697988 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:53.697925 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:01:53.698462 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:53.697997 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:01:53.698462 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:53.698063 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:01:53.698462 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.698080 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:53.698462 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.698091 2567 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:01:53.698462 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.698110 2567 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585cd67447-r2kt6: secret "image-registry-tls" not found May 26 16:01:53.698462 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.698148 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls podName:70fb5ba8-e654-4f66-8a9a-597dec2fde25 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:25.698132487 +0000 UTC m=+98.260865413 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls") pod "dns-default-2kl9l" (UID: "70fb5ba8-e654-4f66-8a9a-597dec2fde25") : secret "dns-default-metrics-tls" not found May 26 16:01:53.698462 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.698166 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls podName:712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd nodeName:}" failed. No retries permitted until 2026-05-26 16:02:25.698160576 +0000 UTC m=+98.260893497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls") pod "image-registry-585cd67447-r2kt6" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd") : secret "image-registry-tls" not found May 26 16:01:53.698462 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.698219 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found May 26 16:01:53.698462 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.698283 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert podName:b8aee9c3-623c-4438-934c-ea0e437eda7a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:25.698269537 +0000 UTC m=+98.261002458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-dwthz" (UID: "b8aee9c3-623c-4438-934c-ea0e437eda7a") : secret "networking-console-plugin-cert" not found May 26 16:01:53.798637 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:53.798605 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:01:53.798637 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:53.798650 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:01:53.798862 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.798744 2567 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:53.798862 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.798795 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert podName:86cf44b2-9266-4e01-a88a-b1552e2ae1bd nodeName:}" failed. No retries permitted until 2026-05-26 16:02:25.798783023 +0000 UTC m=+98.361515956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert") pod "ingress-canary-rl5nn" (UID: "86cf44b2-9266-4e01-a88a-b1552e2ae1bd") : secret "canary-serving-cert" not found May 26 16:01:53.801195 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:53.801176 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 26 16:01:53.809231 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.809213 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found May 26 16:01:53.809326 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:01:53.809316 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs podName:ddf6bd23-ce8a-4927-b11f-c552d481bc33 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:57.809299342 +0000 UTC m=+130.372032264 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs") pod "network-metrics-daemon-x7p82" (UID: "ddf6bd23-ce8a-4927-b11f-c552d481bc33") : secret "metrics-daemon-secret" not found May 26 16:01:53.899389 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:53.899354 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:53.901730 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:53.901712 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 26 16:01:53.913632 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:53.913597 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3874d0e8-75c4-4fed-be1a-f305a44abfa8-original-pull-secret\") pod \"global-pull-secret-syncer-6nfnx\" (UID: \"3874d0e8-75c4-4fed-be1a-f305a44abfa8\") " pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:54.000794 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:54.000719 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:54.003584 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:54.003565 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 26 16:01:54.013256 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:54.013229 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 26 16:01:54.024392 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:54.024360 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssvk\" (UniqueName: \"kubernetes.io/projected/86a247ed-5b25-4bd5-86f5-4ebca2a54f44-kube-api-access-nssvk\") pod \"network-check-target-r6sp2\" (UID: \"86a247ed-5b25-4bd5-86f5-4ebca2a54f44\") " pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:54.172526 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:54.172498 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nfnx" May 26 16:01:54.180489 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:54.180462 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45wxn\"" May 26 16:01:54.189390 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:54.189364 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:54.298611 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:54.298386 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6nfnx"] May 26 16:01:54.301027 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:01:54.301000 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3874d0e8_75c4_4fed_be1a_f305a44abfa8.slice/crio-240564ce40414b425c1a5f0c86e9fd3fd599912d5530085203b6d11014a1c8a3 WatchSource:0}: Error finding container 240564ce40414b425c1a5f0c86e9fd3fd599912d5530085203b6d11014a1c8a3: Status 404 returned error can't find the container with id 240564ce40414b425c1a5f0c86e9fd3fd599912d5530085203b6d11014a1c8a3 May 26 16:01:54.314385 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:54.314360 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r6sp2"] May 26 16:01:54.317280 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:01:54.317254 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a247ed_5b25_4bd5_86f5_4ebca2a54f44.slice/crio-1385443332d29aee2c2bf7ad5d58994674839e23c7b953f20c820f0c0a52b770 WatchSource:0}: Error finding container 1385443332d29aee2c2bf7ad5d58994674839e23c7b953f20c820f0c0a52b770: Status 404 returned error can't find the container with id 1385443332d29aee2c2bf7ad5d58994674839e23c7b953f20c820f0c0a52b770 May 26 16:01:55.297588 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:55.297544 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r6sp2" event={"ID":"86a247ed-5b25-4bd5-86f5-4ebca2a54f44","Type":"ContainerStarted","Data":"1385443332d29aee2c2bf7ad5d58994674839e23c7b953f20c820f0c0a52b770"} May 26 16:01:55.299258 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:55.299230 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6nfnx" event={"ID":"3874d0e8-75c4-4fed-be1a-f305a44abfa8","Type":"ContainerStarted","Data":"240564ce40414b425c1a5f0c86e9fd3fd599912d5530085203b6d11014a1c8a3"} May 26 16:01:59.308690 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:59.308656 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r6sp2" event={"ID":"86a247ed-5b25-4bd5-86f5-4ebca2a54f44","Type":"ContainerStarted","Data":"21ed2d5a1840944f4ae4580891f77582a2ac24c9348c51c1a0ca03e0da322bbc"} May 26 16:01:59.309055 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:59.308725 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:01:59.322893 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:01:59.322851 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r6sp2" podStartSLOduration=66.473347225 podStartE2EDuration="1m11.322836144s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:01:54.318855557 +0000 UTC m=+66.881588478" lastFinishedPulling="2026-05-26 16:01:59.168344473 +0000 UTC m=+71.731077397" observedRunningTime="2026-05-26 16:01:59.321918952 +0000 UTC m=+71.884651897" watchObservedRunningTime="2026-05-26 16:01:59.322836144 +0000 UTC m=+71.885569087" May 26 16:02:00.313289 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:02:00.313252 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6nfnx" event={"ID":"3874d0e8-75c4-4fed-be1a-f305a44abfa8","Type":"ContainerStarted","Data":"507a76f48ebe23c9ec6e23b53208b9f15b82ed4765376efed79cf9ce37173f8d"} May 26 16:02:00.328164 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:02:00.328120 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6nfnx" podStartSLOduration=66.447954942 podStartE2EDuration="1m11.328107796s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:01:54.302637406 +0000 UTC m=+66.865370326" lastFinishedPulling="2026-05-26 16:01:59.182790256 +0000 UTC m=+71.745523180" observedRunningTime="2026-05-26 16:02:00.327388153 +0000 UTC m=+72.890121096" watchObservedRunningTime="2026-05-26 16:02:00.328107796 +0000 UTC m=+72.890840738" May 26 16:02:25.760040 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:02:25.759998 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:02:25.760382 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:02:25.760078 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:02:25.760382 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:02:25.760101 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:02:25.760382 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:25.760148 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found May 26 16:02:25.760382 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:25.760196 2567 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:02:25.760382 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:25.760211 2567 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585cd67447-r2kt6: secret "image-registry-tls" not found May 26 16:02:25.760382 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:25.760214 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:02:25.760382 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:25.760218 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert podName:b8aee9c3-623c-4438-934c-ea0e437eda7a nodeName:}" failed. No retries permitted until 2026-05-26 16:03:29.760203567 +0000 UTC m=+162.322936487 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-dwthz" (UID: "b8aee9c3-623c-4438-934c-ea0e437eda7a") : secret "networking-console-plugin-cert" not found May 26 16:02:25.760382 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:25.760275 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls podName:712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd nodeName:}" failed. No retries permitted until 2026-05-26 16:03:29.760259154 +0000 UTC m=+162.322992078 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls") pod "image-registry-585cd67447-r2kt6" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd") : secret "image-registry-tls" not found May 26 16:02:25.760382 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:25.760296 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls podName:70fb5ba8-e654-4f66-8a9a-597dec2fde25 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:29.760286513 +0000 UTC m=+162.323019436 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls") pod "dns-default-2kl9l" (UID: "70fb5ba8-e654-4f66-8a9a-597dec2fde25") : secret "dns-default-metrics-tls" not found May 26 16:02:25.861119 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:02:25.861076 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:02:25.861249 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:25.861224 2567 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:02:25.861294 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:25.861285 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert podName:86cf44b2-9266-4e01-a88a-b1552e2ae1bd nodeName:}" failed. No retries permitted until 2026-05-26 16:03:29.861270336 +0000 UTC m=+162.424003257 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert") pod "ingress-canary-rl5nn" (UID: "86cf44b2-9266-4e01-a88a-b1552e2ae1bd") : secret "canary-serving-cert" not found May 26 16:02:30.315709 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:02:30.315675 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r6sp2" May 26 16:02:57.906115 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:02:57.906071 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:02:57.906587 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:57.906194 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found May 26 16:02:57.906587 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:02:57.906249 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs podName:ddf6bd23-ce8a-4927-b11f-c552d481bc33 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:59.906232723 +0000 UTC m=+252.468965648 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs") pod "network-metrics-daemon-x7p82" (UID: "ddf6bd23-ce8a-4927-b11f-c552d481bc33") : secret "metrics-daemon-secret" not found May 26 16:03:24.775947 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:24.775896 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" podUID="b8aee9c3-623c-4438-934c-ea0e437eda7a" May 26 16:03:24.810283 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:24.810240 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-585cd67447-r2kt6" podUID="712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" May 26 16:03:24.861460 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:24.861410 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2kl9l" podUID="70fb5ba8-e654-4f66-8a9a-597dec2fde25" May 26 16:03:24.876568 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:24.876534 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-rl5nn" podUID="86cf44b2-9266-4e01-a88a-b1552e2ae1bd" May 26 16:03:25.066333 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:25.066233 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-x7p82" podUID="ddf6bd23-ce8a-4927-b11f-c552d481bc33" May 26 16:03:25.509274 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:25.509242 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:03:25.509476 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:25.509242 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2kl9l" May 26 16:03:25.509476 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:25.509242 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:03:29.845258 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:29.845205 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:03:29.845258 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:29.845265 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") pod \"image-registry-585cd67447-r2kt6\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:03:29.845764 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:29.845302 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:03:29.845764 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:29.845373 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:03:29.845764 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:29.845446 2567 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:03:29.845764 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:29.845379 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found May 26 16:03:29.845764 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:29.845453 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls podName:70fb5ba8-e654-4f66-8a9a-597dec2fde25 nodeName:}" failed. No retries permitted until 2026-05-26 16:05:31.845435797 +0000 UTC m=+284.408168717 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls") pod "dns-default-2kl9l" (UID: "70fb5ba8-e654-4f66-8a9a-597dec2fde25") : secret "dns-default-metrics-tls" not found May 26 16:03:29.845764 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:29.845531 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert podName:b8aee9c3-623c-4438-934c-ea0e437eda7a nodeName:}" failed. No retries permitted until 2026-05-26 16:05:31.845518996 +0000 UTC m=+284.408251917 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-dwthz" (UID: "b8aee9c3-623c-4438-934c-ea0e437eda7a") : secret "networking-console-plugin-cert" not found May 26 16:03:29.845764 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:29.845468 2567 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585cd67447-r2kt6: secret "image-registry-tls" not found May 26 16:03:29.845764 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:29.845554 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls podName:712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd nodeName:}" failed. No retries permitted until 2026-05-26 16:05:31.84554799 +0000 UTC m=+284.408280911 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls") pod "image-registry-585cd67447-r2kt6" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd") : secret "image-registry-tls" not found May 26 16:03:29.946134 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:29.946102 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:03:29.946275 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:29.946243 2567 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:03:29.946325 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:03:29.946314 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert podName:86cf44b2-9266-4e01-a88a-b1552e2ae1bd nodeName:}" failed. No retries permitted until 2026-05-26 16:05:31.946297275 +0000 UTC m=+284.509030200 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert") pod "ingress-canary-rl5nn" (UID: "86cf44b2-9266-4e01-a88a-b1552e2ae1bd") : secret "canary-serving-cert" not found May 26 16:03:32.526880 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:32.526848 2567 generic.go:358] "Generic (PLEG): container finished" podID="35b7a055-0f8a-49b5-86f8-dd00b288fb0b" containerID="46b2b0a46b6421ddf1319ff5143e4d949951b88be4bb3203f7596012917fafa4" exitCode=1 May 26 16:03:32.527355 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:32.526922 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" event={"ID":"35b7a055-0f8a-49b5-86f8-dd00b288fb0b","Type":"ContainerDied","Data":"46b2b0a46b6421ddf1319ff5143e4d949951b88be4bb3203f7596012917fafa4"} May 26 16:03:32.527355 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:32.527337 2567 scope.go:117] "RemoveContainer" containerID="46b2b0a46b6421ddf1319ff5143e4d949951b88be4bb3203f7596012917fafa4" May 26 16:03:32.528180 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:32.528162 2567 generic.go:358] "Generic (PLEG): container finished" podID="0e7021e8-c7c5-426d-8214-1a923d10e9fd" containerID="6acfc8f70af508bf96a823ca6b1cbb92f7381881d4aae8b966a8556064499977" exitCode=255 May 26 16:03:32.528236 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:32.528191 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" event={"ID":"0e7021e8-c7c5-426d-8214-1a923d10e9fd","Type":"ContainerDied","Data":"6acfc8f70af508bf96a823ca6b1cbb92f7381881d4aae8b966a8556064499977"} May 26 16:03:32.528454 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:32.528441 2567 scope.go:117] "RemoveContainer" containerID="6acfc8f70af508bf96a823ca6b1cbb92f7381881d4aae8b966a8556064499977" May 26 16:03:33.247596 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:33.247561 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:03:33.533554 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:33.533471 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9b665b78-6h4qd" event={"ID":"0e7021e8-c7c5-426d-8214-1a923d10e9fd","Type":"ContainerStarted","Data":"1b111af48d2f51ac5d8fdfc045435967889d943e985729281e2df149ca74ae83"} May 26 16:03:33.534972 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:33.534950 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" event={"ID":"35b7a055-0f8a-49b5-86f8-dd00b288fb0b","Type":"ContainerStarted","Data":"27d86b2516e182158db0a796db5277965009656654d773aa25d0a3859f89458c"} May 26 16:03:33.535132 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:33.535122 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:03:33.535714 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:33.535696 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b955d89fd-cw5tq" May 26 16:03:37.045851 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:37.045810 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:03:38.792250 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:38.792221 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-czhdz_3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe/dns-node-resolver/0.log" May 26 16:03:39.045387 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:39.045285 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:03:40.190570 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:03:40.190547 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q7xm7_77d4a48a-9687-4338-b49d-1219168b52e2/node-ca/0.log" May 26 16:04:00.378556 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.378519 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mgdrc"] May 26 16:04:00.381685 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.381664 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.384150 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.384130 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" May 26 16:04:00.384460 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.384444 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" May 26 16:04:00.384833 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.384819 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" May 26 16:04:00.384892 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.384874 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" May 26 16:04:00.385148 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.385134 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vtbnj\"" May 26 16:04:00.393035 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.393009 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mgdrc"] May 26 16:04:00.472728 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.472688 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-crio-socket\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.472905 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.472738 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.472905 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.472757 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ztg\" (UniqueName: \"kubernetes.io/projected/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-kube-api-access-57ztg\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.472905 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.472775 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-data-volume\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.472905 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.472824 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.574123 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.574088 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-crio-socket\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.574307 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.574138 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.574307 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.574163 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57ztg\" (UniqueName: \"kubernetes.io/projected/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-kube-api-access-57ztg\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.574307 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.574193 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-data-volume\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.574307 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.574207 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-crio-socket\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.574307 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.574274 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.574599 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.574581 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-data-volume\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.574900 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.574881 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.576755 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.576734 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.582708 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.582684 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ztg\" (UniqueName: \"kubernetes.io/projected/c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5-kube-api-access-57ztg\") pod \"insights-runtime-extractor-mgdrc\" (UID: \"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5\") " pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.689852 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.689769 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mgdrc" May 26 16:04:00.806821 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:00.806790 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mgdrc"] May 26 16:04:00.809888 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:04:00.809864 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bf2db6_ca7f_47b5_bba6_fabedaca7fe5.slice/crio-156db4d143772727f8cc96c4ad6f755167673b72ad6a05dbe7eea0bbf6c6a385 WatchSource:0}: Error finding container 156db4d143772727f8cc96c4ad6f755167673b72ad6a05dbe7eea0bbf6c6a385: Status 404 returned error can't find the container with id 156db4d143772727f8cc96c4ad6f755167673b72ad6a05dbe7eea0bbf6c6a385 May 26 16:04:01.599581 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:01.599552 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mgdrc" event={"ID":"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5","Type":"ContainerStarted","Data":"750a87396fe2bd38f143683b083edffa47e3f612201b0a47336159607476f786"} May 26 16:04:01.599865 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:01.599591 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mgdrc" event={"ID":"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5","Type":"ContainerStarted","Data":"d0cb973e38dceb7f88bc0f5d097a9fd3b8c770384895d556d777a338bae5bb24"} May 26 16:04:01.599865 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:01.599604 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mgdrc" event={"ID":"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5","Type":"ContainerStarted","Data":"156db4d143772727f8cc96c4ad6f755167673b72ad6a05dbe7eea0bbf6c6a385"} May 26 16:04:02.118042 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:02.117983 2567 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" podUID="666c741a-5087-4f81-aee0-991abc9556bc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" May 26 16:04:03.605583 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:03.605548 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mgdrc" event={"ID":"c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5","Type":"ContainerStarted","Data":"e8031bba2e1942b282de16e1c57ec34ac76d56c028f6ea1ad8f5f2afe713c244"} May 26 16:04:03.627583 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:03.627537 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mgdrc" podStartSLOduration=1.507539193 podStartE2EDuration="3.627524049s" podCreationTimestamp="2026-05-26 16:04:00 +0000 UTC" firstStartedPulling="2026-05-26 16:04:00.859466133 +0000 UTC m=+193.422199054" lastFinishedPulling="2026-05-26 16:04:02.979450985 +0000 UTC m=+195.542183910" observedRunningTime="2026-05-26 16:04:03.627259515 +0000 UTC m=+196.189992459" watchObservedRunningTime="2026-05-26 16:04:03.627524049 +0000 UTC m=+196.190256992" May 26 16:04:12.117510 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:12.117469 2567 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" podUID="666c741a-5087-4f81-aee0-991abc9556bc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" May 26 16:04:14.712790 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.712757 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sm94q"] May 26 16:04:14.715961 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.715929 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.717921 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.717897 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" May 26 16:04:14.718194 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.718167 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" May 26 16:04:14.718311 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.718295 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" May 26 16:04:14.718383 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.718354 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" May 26 16:04:14.718437 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.718382 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" May 26 16:04:14.719045 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.719029 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" May 26 16:04:14.719126 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.719062 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g5xd9\"" May 26 16:04:14.785740 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.785713 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d7de434-2277-41a1-bb6c-85e0255218b7-sys\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.785917 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.785754 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d7de434-2277-41a1-bb6c-85e0255218b7-metrics-client-ca\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.785917 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.785775 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.785917 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.785796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.785917 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.785817 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26wt8\" (UniqueName: \"kubernetes.io/projected/0d7de434-2277-41a1-bb6c-85e0255218b7-kube-api-access-26wt8\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.785917 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.785838 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-tls\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.786118 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.785919 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-textfile\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.786118 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.785968 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d7de434-2277-41a1-bb6c-85e0255218b7-root\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.786118 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.786005 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-wtmp\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.886914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.886864 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-26wt8\" (UniqueName: \"kubernetes.io/projected/0d7de434-2277-41a1-bb6c-85e0255218b7-kube-api-access-26wt8\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.886914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.886921 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-tls\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887171 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.886998 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-textfile\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887171 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887031 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d7de434-2277-41a1-bb6c-85e0255218b7-root\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887171 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887085 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-wtmp\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887171 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:04:14.887086 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found May 26 16:04:14.887171 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887126 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d7de434-2277-41a1-bb6c-85e0255218b7-sys\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887171 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887149 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d7de434-2277-41a1-bb6c-85e0255218b7-root\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887424 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:04:14.887177 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-tls podName:0d7de434-2277-41a1-bb6c-85e0255218b7 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:15.387158023 +0000 UTC m=+207.949890950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-tls") pod "node-exporter-sm94q" (UID: "0d7de434-2277-41a1-bb6c-85e0255218b7") : secret "node-exporter-tls" not found May 26 16:04:14.887424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887215 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d7de434-2277-41a1-bb6c-85e0255218b7-metrics-client-ca\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887214 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d7de434-2277-41a1-bb6c-85e0255218b7-sys\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887246 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887279 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887276 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-wtmp\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887396 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-textfile\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887778 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887752 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.887778 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.887769 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d7de434-2277-41a1-bb6c-85e0255218b7-metrics-client-ca\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.889544 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.889522 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:14.895273 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:14.895251 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26wt8\" (UniqueName: \"kubernetes.io/projected/0d7de434-2277-41a1-bb6c-85e0255218b7-kube-api-access-26wt8\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:15.391347 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:15.391300 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-tls\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:15.391530 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:04:15.391455 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found May 26 16:04:15.391581 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:04:15.391533 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-tls podName:0d7de434-2277-41a1-bb6c-85e0255218b7 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:16.391518168 +0000 UTC m=+208.954251089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-tls") pod "node-exporter-sm94q" (UID: "0d7de434-2277-41a1-bb6c-85e0255218b7") : secret "node-exporter-tls" not found May 26 16:04:16.400650 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:16.400595 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-tls\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:16.402873 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:16.402848 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d7de434-2277-41a1-bb6c-85e0255218b7-node-exporter-tls\") pod \"node-exporter-sm94q\" (UID: \"0d7de434-2277-41a1-bb6c-85e0255218b7\") " pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:16.524448 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:16.524398 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sm94q" May 26 16:04:16.533110 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:04:16.533081 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7de434_2277_41a1_bb6c_85e0255218b7.slice/crio-031f6d1724757e4a7c5e00579e3406a18a8b956b5de98cbc1551cc31bdea4cab WatchSource:0}: Error finding container 031f6d1724757e4a7c5e00579e3406a18a8b956b5de98cbc1551cc31bdea4cab: Status 404 returned error can't find the container with id 031f6d1724757e4a7c5e00579e3406a18a8b956b5de98cbc1551cc31bdea4cab May 26 16:04:16.636663 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:16.636624 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm94q" event={"ID":"0d7de434-2277-41a1-bb6c-85e0255218b7","Type":"ContainerStarted","Data":"031f6d1724757e4a7c5e00579e3406a18a8b956b5de98cbc1551cc31bdea4cab"} May 26 16:04:18.642526 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:18.642494 2567 generic.go:358] "Generic (PLEG): container finished" podID="0d7de434-2277-41a1-bb6c-85e0255218b7" containerID="09848ca2225a09548f535619345a19482f4ac82c9f4e782dd6a3bd3c22cf5cc5" exitCode=0 May 26 16:04:18.643059 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:18.642560 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm94q" event={"ID":"0d7de434-2277-41a1-bb6c-85e0255218b7","Type":"ContainerDied","Data":"09848ca2225a09548f535619345a19482f4ac82c9f4e782dd6a3bd3c22cf5cc5"} May 26 16:04:19.646847 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:19.646812 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm94q" event={"ID":"0d7de434-2277-41a1-bb6c-85e0255218b7","Type":"ContainerStarted","Data":"3f5cb199d9434bd2548acd4eba650a50145f161fc65aafb8bb33be9e2ff7c900"} May 26 16:04:19.646847 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:19.646850 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm94q" event={"ID":"0d7de434-2277-41a1-bb6c-85e0255218b7","Type":"ContainerStarted","Data":"724d0aa1d302097a1198d9d6552655f9ad12a440d8c6f337ab7ca2b34af837d3"} May 26 16:04:19.666351 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:19.666304 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sm94q" podStartSLOduration=4.498180177 podStartE2EDuration="5.666291949s" podCreationTimestamp="2026-05-26 16:04:14 +0000 UTC" firstStartedPulling="2026-05-26 16:04:16.535184702 +0000 UTC m=+209.097917623" lastFinishedPulling="2026-05-26 16:04:17.703296464 +0000 UTC m=+210.266029395" observedRunningTime="2026-05-26 16:04:19.665816784 +0000 UTC m=+212.228549728" watchObservedRunningTime="2026-05-26 16:04:19.666291949 +0000 UTC m=+212.229024891" May 26 16:04:22.117583 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:22.117542 2567 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" podUID="666c741a-5087-4f81-aee0-991abc9556bc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" May 26 16:04:22.117978 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:22.117606 2567 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" May 26 16:04:22.118119 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:22.118089 2567 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2e7d87b38e7800e69d200734cf9a1b3402e7b067c6cef2a6938a6592c1961c46"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" containerMessage="Container service-proxy failed liveness probe, will be restarted" May 26 16:04:22.118165 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:22.118151 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" podUID="666c741a-5087-4f81-aee0-991abc9556bc" containerName="service-proxy" containerID="cri-o://2e7d87b38e7800e69d200734cf9a1b3402e7b067c6cef2a6938a6592c1961c46" gracePeriod=30 May 26 16:04:22.655460 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:22.655428 2567 generic.go:358] "Generic (PLEG): container finished" podID="666c741a-5087-4f81-aee0-991abc9556bc" containerID="2e7d87b38e7800e69d200734cf9a1b3402e7b067c6cef2a6938a6592c1961c46" exitCode=2 May 26 16:04:22.655625 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:22.655471 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" event={"ID":"666c741a-5087-4f81-aee0-991abc9556bc","Type":"ContainerDied","Data":"2e7d87b38e7800e69d200734cf9a1b3402e7b067c6cef2a6938a6592c1961c46"} May 26 16:04:22.655625 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:22.655503 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56db54c59-8djvn" event={"ID":"666c741a-5087-4f81-aee0-991abc9556bc","Type":"ContainerStarted","Data":"5dc7f87f00e892b133ac8ae4111e70206f52d477f83c05f71d85a24e742921ef"} May 26 16:04:22.869449 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:22.869415 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-585cd67447-r2kt6"] May 26 16:04:22.869626 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:04:22.869595 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-585cd67447-r2kt6" podUID="712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" May 26 16:04:23.658447 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.658417 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:04:23.662370 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.662351 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:04:23.760567 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.760536 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-installation-pull-secrets\") pod \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " May 26 16:04:23.760719 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.760593 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-trusted-ca\") pod \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " May 26 16:04:23.760719 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.760628 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-certificates\") pod \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " May 26 16:04:23.760719 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.760662 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-bound-sa-token\") pod \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " May 26 16:04:23.760719 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.760688 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-image-registry-private-configuration\") pod \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " May 26 16:04:23.760956 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.760741 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-kube-api-access-p4vdw\") pod \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " May 26 16:04:23.760956 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.760761 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-ca-trust-extracted\") pod \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\" (UID: \"712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd\") " May 26 16:04:23.761064 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.761037 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:23.761128 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.761102 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:23.761209 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.761183 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:04:23.762957 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.762912 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:04:23.763077 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.762967 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:04:23.763141 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.763124 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:04:23.763182 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.763164 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-kube-api-access-p4vdw" (OuterVolumeSpecName: "kube-api-access-p4vdw") pod "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" (UID: "712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd"). InnerVolumeSpecName "kube-api-access-p4vdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:04:23.862315 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.862280 2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-image-registry-private-configuration\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:04:23.862315 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.862309 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-kube-api-access-p4vdw\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:04:23.862315 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.862322 2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-ca-trust-extracted\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:04:23.862542 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.862335 2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-installation-pull-secrets\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:04:23.862542 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.862350 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-trusted-ca\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:04:23.862542 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.862362 2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-certificates\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:04:23.862542 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:23.862373 2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-bound-sa-token\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:04:24.660913 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:24.660883 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585cd67447-r2kt6" May 26 16:04:24.698676 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:24.698649 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-585cd67447-r2kt6"] May 26 16:04:24.702680 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:24.702651 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-585cd67447-r2kt6"] May 26 16:04:24.870746 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:24.870711 2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd-registry-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:04:26.049288 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:26.049213 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd" path="/var/lib/kubelet/pods/712aeb87-9dc9-44ab-a6d9-53e1fd2c08fd/volumes" May 26 16:04:59.933486 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:59.933449 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:04:59.935773 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:04:59.935752 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf6bd23-ce8a-4927-b11f-c552d481bc33-metrics-certs\") pod \"network-metrics-daemon-x7p82\" (UID: \"ddf6bd23-ce8a-4927-b11f-c552d481bc33\") " pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:05:00.149183 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:00.149151 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8ddfl\"" May 26 16:05:00.157199 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:00.157181 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7p82" May 26 16:05:00.277475 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:00.277440 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7p82"] May 26 16:05:00.280551 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:05:00.280525 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf6bd23_ce8a_4927_b11f_c552d481bc33.slice/crio-da165bb41242c2fc7489a1d13cd6abb32e333521412d60a636b3df9c1f75c0b5 WatchSource:0}: Error finding container da165bb41242c2fc7489a1d13cd6abb32e333521412d60a636b3df9c1f75c0b5: Status 404 returned error can't find the container with id da165bb41242c2fc7489a1d13cd6abb32e333521412d60a636b3df9c1f75c0b5 May 26 16:05:00.750908 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:00.750866 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7p82" event={"ID":"ddf6bd23-ce8a-4927-b11f-c552d481bc33","Type":"ContainerStarted","Data":"da165bb41242c2fc7489a1d13cd6abb32e333521412d60a636b3df9c1f75c0b5"} May 26 16:05:01.755022 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:01.754921 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7p82" event={"ID":"ddf6bd23-ce8a-4927-b11f-c552d481bc33","Type":"ContainerStarted","Data":"a63c601c4c19c67a5c4f11ac185fe8547736a3286ead541a68d701180ee953d9"} May 26 16:05:01.755022 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:01.754976 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7p82" event={"ID":"ddf6bd23-ce8a-4927-b11f-c552d481bc33","Type":"ContainerStarted","Data":"3b4395ebe7b974b478780c91ffbf683ce85542dc97a01e4df37c01a88433e2f8"} May 26 16:05:01.775454 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:01.775402 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x7p82" podStartSLOduration=252.639715484 podStartE2EDuration="4m13.775387211s" podCreationTimestamp="2026-05-26 16:00:48 +0000 UTC" firstStartedPulling="2026-05-26 16:05:00.282404324 +0000 UTC m=+252.845137244" lastFinishedPulling="2026-05-26 16:05:01.418076047 +0000 UTC m=+253.980808971" observedRunningTime="2026-05-26 16:05:01.773508374 +0000 UTC m=+254.336241317" watchObservedRunningTime="2026-05-26 16:05:01.775387211 +0000 UTC m=+254.338120154" May 26 16:05:02.045444 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:02.045371 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-czhdz_3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe/dns-node-resolver/0.log" May 26 16:05:28.510115 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:05:28.510062 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" podUID="b8aee9c3-623c-4438-934c-ea0e437eda7a" May 26 16:05:28.510115 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:05:28.510062 2567 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2kl9l" podUID="70fb5ba8-e654-4f66-8a9a-597dec2fde25" May 26 16:05:28.824461 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:28.824378 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:05:28.824653 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:28.824380 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2kl9l" May 26 16:05:31.895238 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:31.895196 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:05:31.895709 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:31.895269 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:05:31.897593 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:31.897573 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70fb5ba8-e654-4f66-8a9a-597dec2fde25-metrics-tls\") pod \"dns-default-2kl9l\" (UID: \"70fb5ba8-e654-4f66-8a9a-597dec2fde25\") " pod="openshift-dns/dns-default-2kl9l" May 26 16:05:31.897732 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:31.897708 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8aee9c3-623c-4438-934c-ea0e437eda7a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dwthz\" (UID: \"b8aee9c3-623c-4438-934c-ea0e437eda7a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:05:31.996045 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:31.996009 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:05:31.998325 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:31.998297 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cf44b2-9266-4e01-a88a-b1552e2ae1bd-cert\") pod \"ingress-canary-rl5nn\" (UID: \"86cf44b2-9266-4e01-a88a-b1552e2ae1bd\") " pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:05:32.128281 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.128240 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tszrn\"" May 26 16:05:32.128832 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.128815 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lppkw\"" May 26 16:05:32.136658 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.136638 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2kl9l" May 26 16:05:32.136730 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.136642 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" May 26 16:05:32.148558 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.148377 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-22s8h\"" May 26 16:05:32.156650 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.156625 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rl5nn" May 26 16:05:32.275030 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.274995 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz"] May 26 16:05:32.276733 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:05:32.276704 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8aee9c3_623c_4438_934c_ea0e437eda7a.slice/crio-2ee6e4bd8cf25b81da334fa9fc4527d479f638d8a7a571b8f8282c4ccc8e84fd WatchSource:0}: Error finding container 2ee6e4bd8cf25b81da334fa9fc4527d479f638d8a7a571b8f8282c4ccc8e84fd: Status 404 returned error can't find the container with id 2ee6e4bd8cf25b81da334fa9fc4527d479f638d8a7a571b8f8282c4ccc8e84fd May 26 16:05:32.293700 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.293671 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2kl9l"] May 26 16:05:32.296594 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:05:32.296560 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fb5ba8_e654_4f66_8a9a_597dec2fde25.slice/crio-40cdb75e94ae075c2c4560e007097c7bfb6681e187039cc07f83f3c1200f46b9 WatchSource:0}: Error finding container 40cdb75e94ae075c2c4560e007097c7bfb6681e187039cc07f83f3c1200f46b9: Status 404 returned error can't find the container with id 40cdb75e94ae075c2c4560e007097c7bfb6681e187039cc07f83f3c1200f46b9 May 26 16:05:32.314242 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.314185 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rl5nn"] May 26 16:05:32.316573 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:05:32.316540 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cf44b2_9266_4e01_a88a_b1552e2ae1bd.slice/crio-320e9fea3bba1eee5dfd7cedf8577bdec3f9b87c5bd17c71d2a450d15b6307ce WatchSource:0}: Error finding container 320e9fea3bba1eee5dfd7cedf8577bdec3f9b87c5bd17c71d2a450d15b6307ce: Status 404 returned error can't find the container with id 320e9fea3bba1eee5dfd7cedf8577bdec3f9b87c5bd17c71d2a450d15b6307ce May 26 16:05:32.835859 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.835815 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rl5nn" event={"ID":"86cf44b2-9266-4e01-a88a-b1552e2ae1bd","Type":"ContainerStarted","Data":"320e9fea3bba1eee5dfd7cedf8577bdec3f9b87c5bd17c71d2a450d15b6307ce"} May 26 16:05:32.838322 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.838275 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2kl9l" event={"ID":"70fb5ba8-e654-4f66-8a9a-597dec2fde25","Type":"ContainerStarted","Data":"40cdb75e94ae075c2c4560e007097c7bfb6681e187039cc07f83f3c1200f46b9"} May 26 16:05:32.839603 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:32.839562 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" event={"ID":"b8aee9c3-623c-4438-934c-ea0e437eda7a","Type":"ContainerStarted","Data":"2ee6e4bd8cf25b81da334fa9fc4527d479f638d8a7a571b8f8282c4ccc8e84fd"} May 26 16:05:34.845311 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:34.845279 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2kl9l" event={"ID":"70fb5ba8-e654-4f66-8a9a-597dec2fde25","Type":"ContainerStarted","Data":"d958a54f69864d91717e8b5f7f30aebd94fbe74b5ff3b9a7ea61c39999a2e003"} May 26 16:05:34.845311 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:34.845315 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2kl9l" event={"ID":"70fb5ba8-e654-4f66-8a9a-597dec2fde25","Type":"ContainerStarted","Data":"d44e13bdf5a2a641e3e1f9ec1ce293f2bac27d3fc725cd710b45efd2a2cf1d1e"} May 26 16:05:34.845805 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:34.845439 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2kl9l" May 26 16:05:34.850263 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:34.846832 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" event={"ID":"b8aee9c3-623c-4438-934c-ea0e437eda7a","Type":"ContainerStarted","Data":"b776496014d49b5315fe43ecf38ffe48bf8b7370a52a8c6c6b37fae1df4847d6"} May 26 16:05:34.851720 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:34.851697 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rl5nn" event={"ID":"86cf44b2-9266-4e01-a88a-b1552e2ae1bd","Type":"ContainerStarted","Data":"ce4fe89cf363d2fa6b369996aa707bfa25858711c0c2b43dd9a1697c5d87d129"} May 26 16:05:34.865881 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:34.865844 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-2kl9l" podStartSLOduration=251.848433606 podStartE2EDuration="4m13.865833083s" podCreationTimestamp="2026-05-26 16:01:21 +0000 UTC" firstStartedPulling="2026-05-26 16:05:32.298206091 +0000 UTC m=+284.860939013" lastFinishedPulling="2026-05-26 16:05:34.315605567 +0000 UTC m=+286.878338490" observedRunningTime="2026-05-26 16:05:34.863890596 +0000 UTC m=+287.426623533" watchObservedRunningTime="2026-05-26 16:05:34.865833083 +0000 UTC m=+287.428566026" May 26 16:05:34.878031 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:34.877994 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rl5nn" podStartSLOduration=251.877071615 podStartE2EDuration="4m13.877984739s" podCreationTimestamp="2026-05-26 16:01:21 +0000 UTC" firstStartedPulling="2026-05-26 16:05:32.318338888 +0000 UTC m=+284.881071809" lastFinishedPulling="2026-05-26 16:05:34.319252005 +0000 UTC m=+286.881984933" observedRunningTime="2026-05-26 16:05:34.877240962 +0000 UTC m=+287.439973908" watchObservedRunningTime="2026-05-26 16:05:34.877984739 +0000 UTC m=+287.440717681" May 26 16:05:34.894112 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:34.894067 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dwthz" podStartSLOduration=270.862495192 podStartE2EDuration="4m32.894054165s" podCreationTimestamp="2026-05-26 16:01:02 +0000 UTC" firstStartedPulling="2026-05-26 16:05:32.278711741 +0000 UTC m=+284.841444668" lastFinishedPulling="2026-05-26 16:05:34.310270716 +0000 UTC m=+286.873003641" observedRunningTime="2026-05-26 16:05:34.892477001 +0000 UTC m=+287.455209945" watchObservedRunningTime="2026-05-26 16:05:34.894054165 +0000 UTC m=+287.456787141" May 26 16:05:44.857306 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:44.857274 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2kl9l" May 26 16:05:47.979519 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:47.979496 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:05:47.979519 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:47.979508 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:05:47.982467 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:05:47.982446 2567 kubelet.go:1628] "Image garbage collection succeeded" May 26 16:10:47.995551 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:10:47.995513 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:10:47.996493 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:10:47.996473 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:11:52.672211 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.672174 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-psc7c"] May 26 16:11:52.677823 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.677786 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:52.679914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.679896 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" May 26 16:11:52.679914 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.679904 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" May 26 16:11:52.680269 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.680253 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" May 26 16:11:52.681045 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.681031 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" May 26 16:11:52.681107 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.681054 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-q5xzz\"" May 26 16:11:52.684856 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.684528 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-psc7c"] May 26 16:11:52.709342 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.709317 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95b8b\" (UniqueName: \"kubernetes.io/projected/40523a1d-2d25-4132-8509-b85e358efa44-kube-api-access-95b8b\") pod \"keda-admission-cf49989db-psc7c\" (UID: \"40523a1d-2d25-4132-8509-b85e358efa44\") " pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:52.709463 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.709356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40523a1d-2d25-4132-8509-b85e358efa44-certificates\") pod \"keda-admission-cf49989db-psc7c\" (UID: \"40523a1d-2d25-4132-8509-b85e358efa44\") " pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:52.810667 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.810638 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-95b8b\" (UniqueName: \"kubernetes.io/projected/40523a1d-2d25-4132-8509-b85e358efa44-kube-api-access-95b8b\") pod \"keda-admission-cf49989db-psc7c\" (UID: \"40523a1d-2d25-4132-8509-b85e358efa44\") " pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:52.810797 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.810679 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40523a1d-2d25-4132-8509-b85e358efa44-certificates\") pod \"keda-admission-cf49989db-psc7c\" (UID: \"40523a1d-2d25-4132-8509-b85e358efa44\") " pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:52.810797 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:11:52.810771 2567 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found May 26 16:11:52.810797 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:11:52.810788 2567 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-psc7c: secret "keda-admission-webhooks-certs" not found May 26 16:11:52.810913 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:11:52.810843 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40523a1d-2d25-4132-8509-b85e358efa44-certificates podName:40523a1d-2d25-4132-8509-b85e358efa44 nodeName:}" failed. No retries permitted until 2026-05-26 16:11:53.310827508 +0000 UTC m=+665.873560430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/40523a1d-2d25-4132-8509-b85e358efa44-certificates") pod "keda-admission-cf49989db-psc7c" (UID: "40523a1d-2d25-4132-8509-b85e358efa44") : secret "keda-admission-webhooks-certs" not found May 26 16:11:52.820028 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:52.819995 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95b8b\" (UniqueName: \"kubernetes.io/projected/40523a1d-2d25-4132-8509-b85e358efa44-kube-api-access-95b8b\") pod \"keda-admission-cf49989db-psc7c\" (UID: \"40523a1d-2d25-4132-8509-b85e358efa44\") " pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:53.315014 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:53.314976 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40523a1d-2d25-4132-8509-b85e358efa44-certificates\") pod \"keda-admission-cf49989db-psc7c\" (UID: \"40523a1d-2d25-4132-8509-b85e358efa44\") " pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:53.317431 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:53.317411 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40523a1d-2d25-4132-8509-b85e358efa44-certificates\") pod \"keda-admission-cf49989db-psc7c\" (UID: \"40523a1d-2d25-4132-8509-b85e358efa44\") " pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:53.588319 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:53.588236 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:53.709056 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:53.709026 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-psc7c"] May 26 16:11:53.712081 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:11:53.712052 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40523a1d_2d25_4132_8509_b85e358efa44.slice/crio-43032caa88cc192a211009c99ef708c847a2207bc6218b2462b494686ab3fa3a WatchSource:0}: Error finding container 43032caa88cc192a211009c99ef708c847a2207bc6218b2462b494686ab3fa3a: Status 404 returned error can't find the container with id 43032caa88cc192a211009c99ef708c847a2207bc6218b2462b494686ab3fa3a May 26 16:11:53.713252 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:53.713235 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:11:53.792099 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:53.792063 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-psc7c" event={"ID":"40523a1d-2d25-4132-8509-b85e358efa44","Type":"ContainerStarted","Data":"43032caa88cc192a211009c99ef708c847a2207bc6218b2462b494686ab3fa3a"} May 26 16:11:56.801151 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:56.801113 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-psc7c" event={"ID":"40523a1d-2d25-4132-8509-b85e358efa44","Type":"ContainerStarted","Data":"92f08b2b3ab556afb58d5b50d3729dd0afd8b9d69949cf2838ad192c90e4ce38"} May 26 16:11:56.801573 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:56.801226 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:11:56.817308 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:11:56.817230 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-psc7c" podStartSLOduration=1.9283968379999998 podStartE2EDuration="4.817216501s" podCreationTimestamp="2026-05-26 16:11:52 +0000 UTC" firstStartedPulling="2026-05-26 16:11:53.713359348 +0000 UTC m=+666.276092269" lastFinishedPulling="2026-05-26 16:11:56.602179011 +0000 UTC m=+669.164911932" observedRunningTime="2026-05-26 16:11:56.816313353 +0000 UTC m=+669.379046302" watchObservedRunningTime="2026-05-26 16:11:56.817216501 +0000 UTC m=+669.379949443" May 26 16:12:17.806851 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:17.806815 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-psc7c" May 26 16:12:56.461861 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.461826 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-6tcpl"] May 26 16:12:56.464812 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.464797 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:56.467629 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.467609 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" May 26 16:12:56.467629 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.467616 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-8t8mv\"" May 26 16:12:56.468524 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.468503 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" May 26 16:12:56.468619 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.468535 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" May 26 16:12:56.477157 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.477135 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-6tcpl"] May 26 16:12:56.564974 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.564919 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cf9f\" (UniqueName: \"kubernetes.io/projected/85fe22d6-a6b2-4d67-9077-465a9566546f-kube-api-access-8cf9f\") pod \"kserve-controller-manager-6f7545b5b9-6tcpl\" (UID: \"85fe22d6-a6b2-4d67-9077-465a9566546f\") " pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:56.565146 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.564986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85fe22d6-a6b2-4d67-9077-465a9566546f-cert\") pod \"kserve-controller-manager-6f7545b5b9-6tcpl\" (UID: \"85fe22d6-a6b2-4d67-9077-465a9566546f\") " pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:56.666179 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.666141 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85fe22d6-a6b2-4d67-9077-465a9566546f-cert\") pod \"kserve-controller-manager-6f7545b5b9-6tcpl\" (UID: \"85fe22d6-a6b2-4d67-9077-465a9566546f\") " pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:56.666365 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:12:56.666291 2567 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found May 26 16:12:56.666365 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.666289 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8cf9f\" (UniqueName: \"kubernetes.io/projected/85fe22d6-a6b2-4d67-9077-465a9566546f-kube-api-access-8cf9f\") pod \"kserve-controller-manager-6f7545b5b9-6tcpl\" (UID: \"85fe22d6-a6b2-4d67-9077-465a9566546f\") " pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:56.666473 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:12:56.666357 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85fe22d6-a6b2-4d67-9077-465a9566546f-cert podName:85fe22d6-a6b2-4d67-9077-465a9566546f nodeName:}" failed. No retries permitted until 2026-05-26 16:12:57.166339183 +0000 UTC m=+729.729072105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/85fe22d6-a6b2-4d67-9077-465a9566546f-cert") pod "kserve-controller-manager-6f7545b5b9-6tcpl" (UID: "85fe22d6-a6b2-4d67-9077-465a9566546f") : secret "kserve-webhook-server-cert" not found May 26 16:12:56.680015 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:56.679990 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cf9f\" (UniqueName: \"kubernetes.io/projected/85fe22d6-a6b2-4d67-9077-465a9566546f-kube-api-access-8cf9f\") pod \"kserve-controller-manager-6f7545b5b9-6tcpl\" (UID: \"85fe22d6-a6b2-4d67-9077-465a9566546f\") " pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:57.171542 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:57.171500 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85fe22d6-a6b2-4d67-9077-465a9566546f-cert\") pod \"kserve-controller-manager-6f7545b5b9-6tcpl\" (UID: \"85fe22d6-a6b2-4d67-9077-465a9566546f\") " pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:57.173833 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:57.173802 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85fe22d6-a6b2-4d67-9077-465a9566546f-cert\") pod \"kserve-controller-manager-6f7545b5b9-6tcpl\" (UID: \"85fe22d6-a6b2-4d67-9077-465a9566546f\") " pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:57.374840 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:57.374804 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:57.493999 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:57.493915 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-6tcpl"] May 26 16:12:57.497367 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:12:57.497340 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fe22d6_a6b2_4d67_9077_465a9566546f.slice/crio-336b6eafff700317d1417b14db744f88157247aa0e739a2f5818c207a76a83f5 WatchSource:0}: Error finding container 336b6eafff700317d1417b14db744f88157247aa0e739a2f5818c207a76a83f5: Status 404 returned error can't find the container with id 336b6eafff700317d1417b14db744f88157247aa0e739a2f5818c207a76a83f5 May 26 16:12:57.957618 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:57.957577 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" event={"ID":"85fe22d6-a6b2-4d67-9077-465a9566546f","Type":"ContainerStarted","Data":"336b6eafff700317d1417b14db744f88157247aa0e739a2f5818c207a76a83f5"} May 26 16:12:59.964336 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:59.964308 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" event={"ID":"85fe22d6-a6b2-4d67-9077-465a9566546f","Type":"ContainerStarted","Data":"454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c"} May 26 16:12:59.964652 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:59.964384 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:12:59.979904 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:12:59.979737 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" podStartSLOduration=1.58086962 podStartE2EDuration="3.979718219s" podCreationTimestamp="2026-05-26 16:12:56 +0000 UTC" firstStartedPulling="2026-05-26 16:12:57.49869586 +0000 UTC m=+730.061428784" lastFinishedPulling="2026-05-26 16:12:59.897544463 +0000 UTC m=+732.460277383" observedRunningTime="2026-05-26 16:12:59.978289503 +0000 UTC m=+732.541022446" watchObservedRunningTime="2026-05-26 16:12:59.979718219 +0000 UTC m=+732.542451163" May 26 16:13:30.972855 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:30.972819 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:13:34.089513 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.089478 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-6tcpl"] May 26 16:13:34.089888 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.089703 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" podUID="85fe22d6-a6b2-4d67-9077-465a9566546f" containerName="manager" containerID="cri-o://454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c" gracePeriod=10 May 26 16:13:34.109064 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.109036 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-j9b4s"] May 26 16:13:34.111551 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.111533 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:13:34.121030 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.121008 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-j9b4s"] May 26 16:13:34.229732 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.229697 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8d4\" (UniqueName: \"kubernetes.io/projected/1c3ccead-1026-4038-bdae-b92ff3744e7c-kube-api-access-cm8d4\") pod \"kserve-controller-manager-6f7545b5b9-j9b4s\" (UID: \"1c3ccead-1026-4038-bdae-b92ff3744e7c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:13:34.229889 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.229761 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c3ccead-1026-4038-bdae-b92ff3744e7c-cert\") pod \"kserve-controller-manager-6f7545b5b9-j9b4s\" (UID: \"1c3ccead-1026-4038-bdae-b92ff3744e7c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:13:34.321112 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.321091 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:13:34.330873 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.330849 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8d4\" (UniqueName: \"kubernetes.io/projected/1c3ccead-1026-4038-bdae-b92ff3744e7c-kube-api-access-cm8d4\") pod \"kserve-controller-manager-6f7545b5b9-j9b4s\" (UID: \"1c3ccead-1026-4038-bdae-b92ff3744e7c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:13:34.331001 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.330890 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c3ccead-1026-4038-bdae-b92ff3744e7c-cert\") pod \"kserve-controller-manager-6f7545b5b9-j9b4s\" (UID: \"1c3ccead-1026-4038-bdae-b92ff3744e7c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:13:34.333231 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.333212 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c3ccead-1026-4038-bdae-b92ff3744e7c-cert\") pod \"kserve-controller-manager-6f7545b5b9-j9b4s\" (UID: \"1c3ccead-1026-4038-bdae-b92ff3744e7c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:13:34.342657 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.342584 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8d4\" (UniqueName: \"kubernetes.io/projected/1c3ccead-1026-4038-bdae-b92ff3744e7c-kube-api-access-cm8d4\") pod \"kserve-controller-manager-6f7545b5b9-j9b4s\" (UID: \"1c3ccead-1026-4038-bdae-b92ff3744e7c\") " pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:13:34.431828 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.431797 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85fe22d6-a6b2-4d67-9077-465a9566546f-cert\") pod \"85fe22d6-a6b2-4d67-9077-465a9566546f\" (UID: \"85fe22d6-a6b2-4d67-9077-465a9566546f\") " May 26 16:13:34.432003 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.431835 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cf9f\" (UniqueName: \"kubernetes.io/projected/85fe22d6-a6b2-4d67-9077-465a9566546f-kube-api-access-8cf9f\") pod \"85fe22d6-a6b2-4d67-9077-465a9566546f\" (UID: \"85fe22d6-a6b2-4d67-9077-465a9566546f\") " May 26 16:13:34.433968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.433919 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fe22d6-a6b2-4d67-9077-465a9566546f-kube-api-access-8cf9f" (OuterVolumeSpecName: "kube-api-access-8cf9f") pod "85fe22d6-a6b2-4d67-9077-465a9566546f" (UID: "85fe22d6-a6b2-4d67-9077-465a9566546f"). InnerVolumeSpecName "kube-api-access-8cf9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:13:34.433968 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.433957 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85fe22d6-a6b2-4d67-9077-465a9566546f-cert" (OuterVolumeSpecName: "cert") pod "85fe22d6-a6b2-4d67-9077-465a9566546f" (UID: "85fe22d6-a6b2-4d67-9077-465a9566546f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:13:34.466525 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.466500 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:13:34.533203 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.533174 2567 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85fe22d6-a6b2-4d67-9077-465a9566546f-cert\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:13:34.533203 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.533204 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cf9f\" (UniqueName: \"kubernetes.io/projected/85fe22d6-a6b2-4d67-9077-465a9566546f-kube-api-access-8cf9f\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:13:34.585155 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:34.585124 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-j9b4s"] May 26 16:13:34.588091 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:13:34.588067 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3ccead_1026_4038_bdae_b92ff3744e7c.slice/crio-b35e231782bccf7923efa6a7a736ac5ac6f0ecef193727493087485f6a409258 WatchSource:0}: Error finding container b35e231782bccf7923efa6a7a736ac5ac6f0ecef193727493087485f6a409258: Status 404 returned error can't find the container with id b35e231782bccf7923efa6a7a736ac5ac6f0ecef193727493087485f6a409258 May 26 16:13:35.056485 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.056458 2567 generic.go:358] "Generic (PLEG): container finished" podID="85fe22d6-a6b2-4d67-9077-465a9566546f" containerID="454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c" exitCode=0 May 26 16:13:35.056592 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.056510 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" May 26 16:13:35.056592 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.056530 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" event={"ID":"85fe22d6-a6b2-4d67-9077-465a9566546f","Type":"ContainerDied","Data":"454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c"} May 26 16:13:35.056592 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.056562 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-6tcpl" event={"ID":"85fe22d6-a6b2-4d67-9077-465a9566546f","Type":"ContainerDied","Data":"336b6eafff700317d1417b14db744f88157247aa0e739a2f5818c207a76a83f5"} May 26 16:13:35.056592 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.056579 2567 scope.go:117] "RemoveContainer" containerID="454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c" May 26 16:13:35.057678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.057648 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" event={"ID":"1c3ccead-1026-4038-bdae-b92ff3744e7c","Type":"ContainerStarted","Data":"b35e231782bccf7923efa6a7a736ac5ac6f0ecef193727493087485f6a409258"} May 26 16:13:35.077688 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.077671 2567 scope.go:117] "RemoveContainer" containerID="454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c" May 26 16:13:35.078023 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:13:35.078001 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c\": container with ID starting with 454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c not found: ID does not exist" containerID="454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c" May 26 16:13:35.078119 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.078032 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c"} err="failed to get container status \"454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c\": rpc error: code = NotFound desc = could not find container \"454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c\": container with ID starting with 454a8eb3eb955dd7f49a2202c240c6dfe61df92bc474141d50e5bad93e344c4c not found: ID does not exist" May 26 16:13:35.087709 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.087679 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-6tcpl"] May 26 16:13:35.091783 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:35.091763 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-6tcpl"] May 26 16:13:36.049390 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:36.049358 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85fe22d6-a6b2-4d67-9077-465a9566546f" path="/var/lib/kubelet/pods/85fe22d6-a6b2-4d67-9077-465a9566546f/volumes" May 26 16:13:36.061395 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:36.061363 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" event={"ID":"1c3ccead-1026-4038-bdae-b92ff3744e7c","Type":"ContainerStarted","Data":"742fa37a23339bfb50cf5ad845c2ccb9086a71f831e971d73d7c50a1772b6c16"} May 26 16:13:36.061530 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:36.061463 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:13:36.077266 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:13:36.077225 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" podStartSLOduration=1.62247907 podStartE2EDuration="2.077212045s" podCreationTimestamp="2026-05-26 16:13:34 +0000 UTC" firstStartedPulling="2026-05-26 16:13:34.589371411 +0000 UTC m=+767.152104333" lastFinishedPulling="2026-05-26 16:13:35.044104373 +0000 UTC m=+767.606837308" observedRunningTime="2026-05-26 16:13:36.076500476 +0000 UTC m=+768.639233419" watchObservedRunningTime="2026-05-26 16:13:36.077212045 +0000 UTC m=+768.639944982" May 26 16:14:07.069432 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:07.069399 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f7545b5b9-j9b4s" May 26 16:14:07.992441 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:07.992408 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-gxx84"] May 26 16:14:07.992711 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:07.992697 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85fe22d6-a6b2-4d67-9077-465a9566546f" containerName="manager" May 26 16:14:07.992756 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:07.992713 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="85fe22d6-a6b2-4d67-9077-465a9566546f" containerName="manager" May 26 16:14:07.992756 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:07.992753 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="85fe22d6-a6b2-4d67-9077-465a9566546f" containerName="manager" May 26 16:14:07.996626 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:07.996605 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:07.998676 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:07.998650 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" May 26 16:14:07.998807 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:07.998774 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-rwvks\"" May 26 16:14:08.004312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:08.004293 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-gxx84"] May 26 16:14:08.063051 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:08.063015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcdb5\" (UniqueName: \"kubernetes.io/projected/82753df3-8c5c-49a6-89a3-bdea2cfd83f8-kube-api-access-dcdb5\") pod \"model-serving-api-86f7b4b499-gxx84\" (UID: \"82753df3-8c5c-49a6-89a3-bdea2cfd83f8\") " pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:08.063213 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:08.063075 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82753df3-8c5c-49a6-89a3-bdea2cfd83f8-tls-certs\") pod \"model-serving-api-86f7b4b499-gxx84\" (UID: \"82753df3-8c5c-49a6-89a3-bdea2cfd83f8\") " pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:08.164155 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:08.164120 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dcdb5\" (UniqueName: \"kubernetes.io/projected/82753df3-8c5c-49a6-89a3-bdea2cfd83f8-kube-api-access-dcdb5\") pod \"model-serving-api-86f7b4b499-gxx84\" (UID: \"82753df3-8c5c-49a6-89a3-bdea2cfd83f8\") " pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:08.164600 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:08.164166 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82753df3-8c5c-49a6-89a3-bdea2cfd83f8-tls-certs\") pod \"model-serving-api-86f7b4b499-gxx84\" (UID: \"82753df3-8c5c-49a6-89a3-bdea2cfd83f8\") " pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:08.164600 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:14:08.164258 2567 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found May 26 16:14:08.164600 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:14:08.164318 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82753df3-8c5c-49a6-89a3-bdea2cfd83f8-tls-certs podName:82753df3-8c5c-49a6-89a3-bdea2cfd83f8 nodeName:}" failed. No retries permitted until 2026-05-26 16:14:08.664301378 +0000 UTC m=+801.227034299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/82753df3-8c5c-49a6-89a3-bdea2cfd83f8-tls-certs") pod "model-serving-api-86f7b4b499-gxx84" (UID: "82753df3-8c5c-49a6-89a3-bdea2cfd83f8") : secret "model-serving-api-tls" not found May 26 16:14:08.178977 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:08.178950 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcdb5\" (UniqueName: \"kubernetes.io/projected/82753df3-8c5c-49a6-89a3-bdea2cfd83f8-kube-api-access-dcdb5\") pod \"model-serving-api-86f7b4b499-gxx84\" (UID: \"82753df3-8c5c-49a6-89a3-bdea2cfd83f8\") " pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:08.669308 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:08.669272 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82753df3-8c5c-49a6-89a3-bdea2cfd83f8-tls-certs\") pod \"model-serving-api-86f7b4b499-gxx84\" (UID: \"82753df3-8c5c-49a6-89a3-bdea2cfd83f8\") " pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:08.671659 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:08.671639 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82753df3-8c5c-49a6-89a3-bdea2cfd83f8-tls-certs\") pod \"model-serving-api-86f7b4b499-gxx84\" (UID: \"82753df3-8c5c-49a6-89a3-bdea2cfd83f8\") " pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:08.907382 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:08.907350 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:09.031062 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:09.031038 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-gxx84"] May 26 16:14:09.033534 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:14:09.033506 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82753df3_8c5c_49a6_89a3_bdea2cfd83f8.slice/crio-84e8faeb886ecf4923e61fd56acc1493dc318966dff667b2de414c8044d23018 WatchSource:0}: Error finding container 84e8faeb886ecf4923e61fd56acc1493dc318966dff667b2de414c8044d23018: Status 404 returned error can't find the container with id 84e8faeb886ecf4923e61fd56acc1493dc318966dff667b2de414c8044d23018 May 26 16:14:09.146206 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:09.146173 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-gxx84" event={"ID":"82753df3-8c5c-49a6-89a3-bdea2cfd83f8","Type":"ContainerStarted","Data":"84e8faeb886ecf4923e61fd56acc1493dc318966dff667b2de414c8044d23018"} May 26 16:14:12.156147 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:12.156108 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-gxx84" event={"ID":"82753df3-8c5c-49a6-89a3-bdea2cfd83f8","Type":"ContainerStarted","Data":"1cf33db252cff303f21830421405faec5faa1b70e43499227a6142df0994314c"} May 26 16:14:12.156594 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:12.156168 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:12.174197 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:12.174136 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-gxx84" podStartSLOduration=3.035055847 podStartE2EDuration="5.174116632s" podCreationTimestamp="2026-05-26 16:14:07 +0000 UTC" firstStartedPulling="2026-05-26 16:14:09.035157402 +0000 UTC m=+801.597890324" lastFinishedPulling="2026-05-26 16:14:11.174218189 +0000 UTC m=+803.736951109" observedRunningTime="2026-05-26 16:14:12.173488834 +0000 UTC m=+804.736221777" watchObservedRunningTime="2026-05-26 16:14:12.174116632 +0000 UTC m=+804.736849575" May 26 16:14:23.162070 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:23.162036 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-gxx84" May 26 16:14:46.061473 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.061432 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7"] May 26 16:14:46.064575 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.064554 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.066845 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.066818 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" May 26 16:14:46.066845 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.066844 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dzvrr\"" May 26 16:14:46.067856 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.067832 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c73f1-predictor-serving-cert\"" May 26 16:14:46.067856 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.067845 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c73f1-kube-rbac-proxy-sar-config\"" May 26 16:14:46.068089 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.067882 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" May 26 16:14:46.072227 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.072206 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7"] May 26 16:14:46.137955 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.137902 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-error-404-isvc-c73f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.138117 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.137961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49f6\" (UniqueName: \"kubernetes.io/projected/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-kube-api-access-h49f6\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.138117 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.138011 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-proxy-tls\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.238338 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.238298 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-error-404-isvc-c73f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.238338 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.238340 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h49f6\" (UniqueName: \"kubernetes.io/projected/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-kube-api-access-h49f6\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.238591 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.238400 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-proxy-tls\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.238591 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:14:46.238505 2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-serving-cert: secret "error-404-isvc-c73f1-predictor-serving-cert" not found May 26 16:14:46.238591 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:14:46.238561 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-proxy-tls podName:4751ea58-7db7-4b7f-b5c9-57e72c3fe07d nodeName:}" failed. No retries permitted until 2026-05-26 16:14:46.738542486 +0000 UTC m=+839.301275406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-proxy-tls") pod "error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" (UID: "4751ea58-7db7-4b7f-b5c9-57e72c3fe07d") : secret "error-404-isvc-c73f1-predictor-serving-cert" not found May 26 16:14:46.239122 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.239098 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-error-404-isvc-c73f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.246477 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.246454 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49f6\" (UniqueName: \"kubernetes.io/projected/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-kube-api-access-h49f6\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.419337 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.419304 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj"] May 26 16:14:46.422836 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.422814 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.425228 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.425206 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" May 26 16:14:46.425320 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.425235 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" May 26 16:14:46.436094 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.436062 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj"] May 26 16:14:46.439162 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.439137 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9476710-d812-4e7d-ae40-bebf65649165-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.439267 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.439175 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9476710-d812-4e7d-ae40-bebf65649165-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.439267 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.439225 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7jdf\" (UniqueName: \"kubernetes.io/projected/f9476710-d812-4e7d-ae40-bebf65649165-kube-api-access-p7jdf\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.439267 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.439242 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9476710-d812-4e7d-ae40-bebf65649165-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.539854 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.539811 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p7jdf\" (UniqueName: \"kubernetes.io/projected/f9476710-d812-4e7d-ae40-bebf65649165-kube-api-access-p7jdf\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.539854 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.539857 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9476710-d812-4e7d-ae40-bebf65649165-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.540136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.539918 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9476710-d812-4e7d-ae40-bebf65649165-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.540136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.539982 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9476710-d812-4e7d-ae40-bebf65649165-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.540136 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:14:46.540037 2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found May 26 16:14:46.540136 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:14:46.540107 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9476710-d812-4e7d-ae40-bebf65649165-proxy-tls podName:f9476710-d812-4e7d-ae40-bebf65649165 nodeName:}" failed. No retries permitted until 2026-05-26 16:14:47.040087759 +0000 UTC m=+839.602820703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f9476710-d812-4e7d-ae40-bebf65649165-proxy-tls") pod "isvc-xgboost-graph-predictor-669d8d6456-dk8qj" (UID: "f9476710-d812-4e7d-ae40-bebf65649165") : secret "isvc-xgboost-graph-predictor-serving-cert" not found May 26 16:14:46.540421 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.540400 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9476710-d812-4e7d-ae40-bebf65649165-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.540681 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.540661 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9476710-d812-4e7d-ae40-bebf65649165-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.548378 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.548347 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7jdf\" (UniqueName: \"kubernetes.io/projected/f9476710-d812-4e7d-ae40-bebf65649165-kube-api-access-p7jdf\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:46.741369 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.741291 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-proxy-tls\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.743689 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.743656 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-proxy-tls\") pod \"error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:46.975749 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:46.975705 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:14:47.044172 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:47.044139 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9476710-d812-4e7d-ae40-bebf65649165-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:47.046850 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:47.046819 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9476710-d812-4e7d-ae40-bebf65649165-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-dk8qj\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:47.104682 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:47.104647 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7"] May 26 16:14:47.112018 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:14:47.111963 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4751ea58_7db7_4b7f_b5c9_57e72c3fe07d.slice/crio-0c1ab16de774b76e9000e961e6e905c37e79b6c4b22e2de7a4d4328300188fa3 WatchSource:0}: Error finding container 0c1ab16de774b76e9000e961e6e905c37e79b6c4b22e2de7a4d4328300188fa3: Status 404 returned error can't find the container with id 0c1ab16de774b76e9000e961e6e905c37e79b6c4b22e2de7a4d4328300188fa3 May 26 16:14:47.244816 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:47.244780 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" event={"ID":"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d","Type":"ContainerStarted","Data":"0c1ab16de774b76e9000e961e6e905c37e79b6c4b22e2de7a4d4328300188fa3"} May 26 16:14:47.337356 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:47.337268 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:14:47.466536 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:47.466481 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj"] May 26 16:14:47.472124 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:14:47.472086 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9476710_d812_4e7d_ae40_bebf65649165.slice/crio-4e9b8f2750d1366f73cc9ed528d5cb8adc167a20532c2e0fa097eaa3271b0065 WatchSource:0}: Error finding container 4e9b8f2750d1366f73cc9ed528d5cb8adc167a20532c2e0fa097eaa3271b0065: Status 404 returned error can't find the container with id 4e9b8f2750d1366f73cc9ed528d5cb8adc167a20532c2e0fa097eaa3271b0065 May 26 16:14:48.249397 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:48.249356 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" event={"ID":"f9476710-d812-4e7d-ae40-bebf65649165","Type":"ContainerStarted","Data":"4e9b8f2750d1366f73cc9ed528d5cb8adc167a20532c2e0fa097eaa3271b0065"} May 26 16:14:52.263798 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:52.263759 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" event={"ID":"f9476710-d812-4e7d-ae40-bebf65649165","Type":"ContainerStarted","Data":"98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd"} May 26 16:14:57.281789 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:57.281349 2567 generic.go:358] "Generic (PLEG): container finished" podID="f9476710-d812-4e7d-ae40-bebf65649165" containerID="98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd" exitCode=0 May 26 16:14:57.281789 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:57.281404 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" event={"ID":"f9476710-d812-4e7d-ae40-bebf65649165","Type":"ContainerDied","Data":"98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd"} May 26 16:14:59.289840 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:14:59.289800 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" event={"ID":"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d","Type":"ContainerStarted","Data":"d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e"} May 26 16:15:03.304961 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:03.304909 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" event={"ID":"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d","Type":"ContainerStarted","Data":"055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51"} May 26 16:15:03.305434 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:03.305170 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:15:03.323310 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:03.323259 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" podStartSLOduration=2.176775018 podStartE2EDuration="17.323244225s" podCreationTimestamp="2026-05-26 16:14:46 +0000 UTC" firstStartedPulling="2026-05-26 16:14:47.113600214 +0000 UTC m=+839.676333135" lastFinishedPulling="2026-05-26 16:15:02.260069407 +0000 UTC m=+854.822802342" observedRunningTime="2026-05-26 16:15:03.321268466 +0000 UTC m=+855.884001412" watchObservedRunningTime="2026-05-26 16:15:03.323244225 +0000 UTC m=+855.885977168" May 26 16:15:04.307982 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:04.307929 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:15:04.309225 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:04.309195 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" May 26 16:15:05.310632 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:05.310595 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" May 26 16:15:10.315995 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:10.315965 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:15:10.316411 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:10.316383 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" May 26 16:15:20.316688 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:20.316646 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" May 26 16:15:23.369722 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:23.369688 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" event={"ID":"f9476710-d812-4e7d-ae40-bebf65649165","Type":"ContainerStarted","Data":"5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf"} May 26 16:15:23.369722 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:23.369727 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" event={"ID":"f9476710-d812-4e7d-ae40-bebf65649165","Type":"ContainerStarted","Data":"e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5"} May 26 16:15:23.370211 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:23.369947 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:15:23.388027 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:23.387978 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podStartSLOduration=2.373630612 podStartE2EDuration="37.387965155s" podCreationTimestamp="2026-05-26 16:14:46 +0000 UTC" firstStartedPulling="2026-05-26 16:14:47.474437976 +0000 UTC m=+840.037170896" lastFinishedPulling="2026-05-26 16:15:22.488772506 +0000 UTC m=+875.051505439" observedRunningTime="2026-05-26 16:15:23.386555656 +0000 UTC m=+875.949288599" watchObservedRunningTime="2026-05-26 16:15:23.387965155 +0000 UTC m=+875.950698098" May 26 16:15:24.372489 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:24.372451 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:15:24.373976 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:24.373923 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" May 26 16:15:25.375659 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:25.375620 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" May 26 16:15:30.316807 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:30.316766 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" May 26 16:15:30.380298 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:30.380266 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:15:30.380812 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:30.380770 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" May 26 16:15:40.316769 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:40.316680 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" May 26 16:15:40.381000 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:40.380962 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" May 26 16:15:48.017337 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:48.017307 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:15:48.018745 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:48.018726 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:15:50.317678 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:50.317653 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:15:50.381544 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:15:50.381500 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" May 26 16:16:00.380744 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:00.380692 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" May 26 16:16:10.381387 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:10.381338 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" May 26 16:16:16.067522 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.067483 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7"] May 26 16:16:16.068009 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.067872 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" containerID="cri-o://d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e" gracePeriod=30 May 26 16:16:16.068009 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.067898 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kube-rbac-proxy" containerID="cri-o://055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51" gracePeriod=30 May 26 16:16:16.250287 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.250256 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8"] May 26 16:16:16.253391 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.253375 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.255434 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.255412 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-df0d2-kube-rbac-proxy-sar-config\"" May 26 16:16:16.255434 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.255424 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-df0d2-predictor-serving-cert\"" May 26 16:16:16.262594 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.262554 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8"] May 26 16:16:16.347651 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.347571 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-proxy-tls\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.347651 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.347620 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4gr\" (UniqueName: \"kubernetes.io/projected/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-kube-api-access-pb4gr\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.347835 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.347691 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-error-404-isvc-df0d2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.449101 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.449067 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-proxy-tls\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.449285 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.449124 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4gr\" (UniqueName: \"kubernetes.io/projected/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-kube-api-access-pb4gr\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.449285 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.449171 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-error-404-isvc-df0d2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.449285 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:16:16.449241 2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-serving-cert: secret "error-404-isvc-df0d2-predictor-serving-cert" not found May 26 16:16:16.449452 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:16:16.449324 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-proxy-tls podName:2fd1085c-0886-4d4d-b4b6-eeca8be643eb nodeName:}" failed. No retries permitted until 2026-05-26 16:16:16.949299766 +0000 UTC m=+929.512032709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-proxy-tls") pod "error-404-isvc-df0d2-predictor-66496f47bf-q49z8" (UID: "2fd1085c-0886-4d4d-b4b6-eeca8be643eb") : secret "error-404-isvc-df0d2-predictor-serving-cert" not found May 26 16:16:16.449838 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.449818 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-error-404-isvc-df0d2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.459691 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.459660 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4gr\" (UniqueName: \"kubernetes.io/projected/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-kube-api-access-pb4gr\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.509136 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.509103 2567 generic.go:358] "Generic (PLEG): container finished" podID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerID="055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51" exitCode=2 May 26 16:16:16.509294 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.509164 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" event={"ID":"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d","Type":"ContainerDied","Data":"055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51"} May 26 16:16:16.952436 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.952398 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-proxy-tls\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:16.954763 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:16.954742 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-proxy-tls\") pod \"error-404-isvc-df0d2-predictor-66496f47bf-q49z8\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:17.163928 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:17.163891 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:17.293332 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:17.293292 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8"] May 26 16:16:17.298378 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:16:17.298353 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd1085c_0886_4d4d_b4b6_eeca8be643eb.slice/crio-1d721289a65bfb407e55b89db4189f7657abf57b4e200f778cf9820c6884fbe3 WatchSource:0}: Error finding container 1d721289a65bfb407e55b89db4189f7657abf57b4e200f778cf9820c6884fbe3: Status 404 returned error can't find the container with id 1d721289a65bfb407e55b89db4189f7657abf57b4e200f778cf9820c6884fbe3 May 26 16:16:17.514203 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:17.514121 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" event={"ID":"2fd1085c-0886-4d4d-b4b6-eeca8be643eb","Type":"ContainerStarted","Data":"a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f"} May 26 16:16:17.514203 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:17.514158 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" event={"ID":"2fd1085c-0886-4d4d-b4b6-eeca8be643eb","Type":"ContainerStarted","Data":"e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea"} May 26 16:16:17.514203 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:17.514167 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" event={"ID":"2fd1085c-0886-4d4d-b4b6-eeca8be643eb","Type":"ContainerStarted","Data":"1d721289a65bfb407e55b89db4189f7657abf57b4e200f778cf9820c6884fbe3"} May 26 16:16:17.514457 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:17.514295 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:17.531126 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:17.531073 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" podStartSLOduration=1.531053956 podStartE2EDuration="1.531053956s" podCreationTimestamp="2026-05-26 16:16:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:16:17.530257809 +0000 UTC m=+930.092990752" watchObservedRunningTime="2026-05-26 16:16:17.531053956 +0000 UTC m=+930.093786900" May 26 16:16:18.516740 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:18.516702 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:18.518030 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:18.518004 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" May 26 16:16:19.314144 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.314122 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:16:19.370473 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.370445 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h49f6\" (UniqueName: \"kubernetes.io/projected/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-kube-api-access-h49f6\") pod \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " May 26 16:16:19.370620 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.370486 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-proxy-tls\") pod \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " May 26 16:16:19.370620 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.370512 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-error-404-isvc-c73f1-kube-rbac-proxy-sar-config\") pod \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\" (UID: \"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d\") " May 26 16:16:19.370920 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.370898 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-error-404-isvc-c73f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-c73f1-kube-rbac-proxy-sar-config") pod "4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" (UID: "4751ea58-7db7-4b7f-b5c9-57e72c3fe07d"). InnerVolumeSpecName "error-404-isvc-c73f1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:16:19.372710 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.372683 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" (UID: "4751ea58-7db7-4b7f-b5c9-57e72c3fe07d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:16:19.373185 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.373161 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-kube-api-access-h49f6" (OuterVolumeSpecName: "kube-api-access-h49f6") pod "4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" (UID: "4751ea58-7db7-4b7f-b5c9-57e72c3fe07d"). InnerVolumeSpecName "kube-api-access-h49f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:16:19.471787 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.471738 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h49f6\" (UniqueName: \"kubernetes.io/projected/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-kube-api-access-h49f6\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:16:19.471787 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.471782 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:16:19.471787 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.471795 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d-error-404-isvc-c73f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:16:19.520158 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.520124 2567 generic.go:358] "Generic (PLEG): container finished" podID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerID="d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e" exitCode=0 May 26 16:16:19.520559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.520200 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" May 26 16:16:19.520559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.520210 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" event={"ID":"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d","Type":"ContainerDied","Data":"d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e"} May 26 16:16:19.520559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.520250 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7" event={"ID":"4751ea58-7db7-4b7f-b5c9-57e72c3fe07d","Type":"ContainerDied","Data":"0c1ab16de774b76e9000e961e6e905c37e79b6c4b22e2de7a4d4328300188fa3"} May 26 16:16:19.520559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.520265 2567 scope.go:117] "RemoveContainer" containerID="055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51" May 26 16:16:19.520777 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.520740 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" May 26 16:16:19.528766 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.528751 2567 scope.go:117] "RemoveContainer" containerID="d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e" May 26 16:16:19.535556 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.535541 2567 scope.go:117] "RemoveContainer" containerID="055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51" May 26 16:16:19.535784 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:16:19.535769 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51\": container with ID starting with 055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51 not found: ID does not exist" containerID="055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51" May 26 16:16:19.535828 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.535791 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51"} err="failed to get container status \"055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51\": rpc error: code = NotFound desc = could not find container \"055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51\": container with ID starting with 055f892d0b74e9071e77a98a25601c73f1bb95e93f6b95495ffe5f9a48517a51 not found: ID does not exist" May 26 16:16:19.535828 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.535805 2567 scope.go:117] "RemoveContainer" containerID="d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e" May 26 16:16:19.536054 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:16:19.536040 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e\": container with ID starting with d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e not found: ID does not exist" containerID="d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e" May 26 16:16:19.536110 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.536056 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e"} err="failed to get container status \"d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e\": rpc error: code = NotFound desc = could not find container \"d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e\": container with ID starting with d93150857cc892e0d425864a44eb0c86fe331dcf2c8ed9fdb6a2b17c8af9b85e not found: ID does not exist" May 26 16:16:19.541537 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.541516 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7"] May 26 16:16:19.547654 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:19.547633 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c73f1-predictor-7c8d9dc697-5sqs7"] May 26 16:16:20.054495 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:20.054459 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" path="/var/lib/kubelet/pods/4751ea58-7db7-4b7f-b5c9-57e72c3fe07d/volumes" May 26 16:16:20.381522 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:20.381471 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" May 26 16:16:24.525445 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:24.525415 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:16:24.525884 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:24.525859 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" May 26 16:16:30.382012 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:30.381976 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:16:34.526526 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:34.526485 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" May 26 16:16:44.526648 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:44.526606 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" May 26 16:16:54.526466 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:54.526426 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" May 26 16:16:56.156810 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.156773 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj"] May 26 16:16:56.157197 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.157109 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" containerID="cri-o://e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5" gracePeriod=30 May 26 16:16:56.157238 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.157131 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kube-rbac-proxy" containerID="cri-o://5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf" gracePeriod=30 May 26 16:16:56.159830 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.159802 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd"] May 26 16:16:56.160334 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.160314 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kube-rbac-proxy" May 26 16:16:56.160334 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.160335 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kube-rbac-proxy" May 26 16:16:56.160506 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.160350 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" May 26 16:16:56.160506 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.160359 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" May 26 16:16:56.160506 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.160430 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kserve-container" May 26 16:16:56.160506 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.160441 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="4751ea58-7db7-4b7f-b5c9-57e72c3fe07d" containerName="kube-rbac-proxy" May 26 16:16:56.163675 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.163656 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.165641 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.165618 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ad62d-predictor-serving-cert\"" May 26 16:16:56.165796 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.165778 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ad62d-kube-rbac-proxy-sar-config\"" May 26 16:16:56.173957 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.173916 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd"] May 26 16:16:56.260466 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.260427 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-proxy-tls\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.260624 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.260489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-error-404-isvc-ad62d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.260624 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.260522 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h72jk\" (UniqueName: \"kubernetes.io/projected/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-kube-api-access-h72jk\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.361460 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.361431 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-proxy-tls\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.361638 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.361472 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-error-404-isvc-ad62d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.361638 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.361501 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h72jk\" (UniqueName: \"kubernetes.io/projected/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-kube-api-access-h72jk\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.361638 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:16:56.361602 2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-serving-cert: secret "error-404-isvc-ad62d-predictor-serving-cert" not found May 26 16:16:56.361781 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:16:56.361676 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-proxy-tls podName:e8cc1d7e-b614-43f6-acbf-7d42248fbbe0 nodeName:}" failed. No retries permitted until 2026-05-26 16:16:56.861652359 +0000 UTC m=+969.424385294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-proxy-tls") pod "error-404-isvc-ad62d-predictor-69787655ff-l67xd" (UID: "e8cc1d7e-b614-43f6-acbf-7d42248fbbe0") : secret "error-404-isvc-ad62d-predictor-serving-cert" not found May 26 16:16:56.362156 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.362139 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-error-404-isvc-ad62d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.369867 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.369843 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72jk\" (UniqueName: \"kubernetes.io/projected/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-kube-api-access-h72jk\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.629169 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.629135 2567 generic.go:358] "Generic (PLEG): container finished" podID="f9476710-d812-4e7d-ae40-bebf65649165" containerID="5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf" exitCode=2 May 26 16:16:56.629332 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.629208 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" event={"ID":"f9476710-d812-4e7d-ae40-bebf65649165","Type":"ContainerDied","Data":"5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf"} May 26 16:16:56.864567 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.864533 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-proxy-tls\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:56.867035 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:56.867017 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-proxy-tls\") pod \"error-404-isvc-ad62d-predictor-69787655ff-l67xd\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:57.079495 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.079410 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:57.203860 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.203827 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd"] May 26 16:16:57.207544 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:16:57.207510 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cc1d7e_b614_43f6_acbf_7d42248fbbe0.slice/crio-0e79c19a393f8f6f619816e88a8a04c798780909486c91a8f5c4e2ded50fa9a1 WatchSource:0}: Error finding container 0e79c19a393f8f6f619816e88a8a04c798780909486c91a8f5c4e2ded50fa9a1: Status 404 returned error can't find the container with id 0e79c19a393f8f6f619816e88a8a04c798780909486c91a8f5c4e2ded50fa9a1 May 26 16:16:57.209185 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.209168 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:16:57.633929 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.633893 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" event={"ID":"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0","Type":"ContainerStarted","Data":"498150941b24224b73570fac55b238c05d28ab7fa66e6aff4dc632f1312e03de"} May 26 16:16:57.633929 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.633929 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" event={"ID":"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0","Type":"ContainerStarted","Data":"772af26713786e488fed44a203bc3c36e6c3407b20f6e813cf774c4953534969"} May 26 16:16:57.633929 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.633960 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" event={"ID":"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0","Type":"ContainerStarted","Data":"0e79c19a393f8f6f619816e88a8a04c798780909486c91a8f5c4e2ded50fa9a1"} May 26 16:16:57.634252 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.633976 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:57.634252 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.633986 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:16:57.635450 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.635424 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" May 26 16:16:57.658023 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:57.657982 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podStartSLOduration=1.6579275390000001 podStartE2EDuration="1.657927539s" podCreationTimestamp="2026-05-26 16:16:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:16:57.656766986 +0000 UTC m=+970.219499928" watchObservedRunningTime="2026-05-26 16:16:57.657927539 +0000 UTC m=+970.220660481" May 26 16:16:58.637183 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:58.637143 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" May 26 16:16:59.801330 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.801308 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:16:59.887343 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.887307 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9476710-d812-4e7d-ae40-bebf65649165-kserve-provision-location\") pod \"f9476710-d812-4e7d-ae40-bebf65649165\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " May 26 16:16:59.887521 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.887361 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9476710-d812-4e7d-ae40-bebf65649165-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"f9476710-d812-4e7d-ae40-bebf65649165\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " May 26 16:16:59.887521 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.887386 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7jdf\" (UniqueName: \"kubernetes.io/projected/f9476710-d812-4e7d-ae40-bebf65649165-kube-api-access-p7jdf\") pod \"f9476710-d812-4e7d-ae40-bebf65649165\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " May 26 16:16:59.887521 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.887413 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9476710-d812-4e7d-ae40-bebf65649165-proxy-tls\") pod \"f9476710-d812-4e7d-ae40-bebf65649165\" (UID: \"f9476710-d812-4e7d-ae40-bebf65649165\") " May 26 16:16:59.887705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.887684 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9476710-d812-4e7d-ae40-bebf65649165-isvc-xgboost-graph-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-kube-rbac-proxy-sar-config") pod "f9476710-d812-4e7d-ae40-bebf65649165" (UID: "f9476710-d812-4e7d-ae40-bebf65649165"). InnerVolumeSpecName "isvc-xgboost-graph-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:16:59.887743 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.887720 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9476710-d812-4e7d-ae40-bebf65649165-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9476710-d812-4e7d-ae40-bebf65649165" (UID: "f9476710-d812-4e7d-ae40-bebf65649165"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:16:59.889559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.889528 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9476710-d812-4e7d-ae40-bebf65649165-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9476710-d812-4e7d-ae40-bebf65649165" (UID: "f9476710-d812-4e7d-ae40-bebf65649165"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:16:59.889656 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.889588 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9476710-d812-4e7d-ae40-bebf65649165-kube-api-access-p7jdf" (OuterVolumeSpecName: "kube-api-access-p7jdf") pod "f9476710-d812-4e7d-ae40-bebf65649165" (UID: "f9476710-d812-4e7d-ae40-bebf65649165"). InnerVolumeSpecName "kube-api-access-p7jdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:16:59.988259 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.988227 2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9476710-d812-4e7d-ae40-bebf65649165-kserve-provision-location\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:16:59.988259 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.988256 2567 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9476710-d812-4e7d-ae40-bebf65649165-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:16:59.988447 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.988272 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7jdf\" (UniqueName: \"kubernetes.io/projected/f9476710-d812-4e7d-ae40-bebf65649165-kube-api-access-p7jdf\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:16:59.988447 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:16:59.988284 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9476710-d812-4e7d-ae40-bebf65649165-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:17:00.648804 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.648771 2567 generic.go:358] "Generic (PLEG): container finished" podID="f9476710-d812-4e7d-ae40-bebf65649165" containerID="e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5" exitCode=0 May 26 16:17:00.648963 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.648833 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" event={"ID":"f9476710-d812-4e7d-ae40-bebf65649165","Type":"ContainerDied","Data":"e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5"} May 26 16:17:00.648963 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.648859 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" event={"ID":"f9476710-d812-4e7d-ae40-bebf65649165","Type":"ContainerDied","Data":"4e9b8f2750d1366f73cc9ed528d5cb8adc167a20532c2e0fa097eaa3271b0065"} May 26 16:17:00.648963 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.648873 2567 scope.go:117] "RemoveContainer" containerID="5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf" May 26 16:17:00.648963 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.648887 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj" May 26 16:17:00.656529 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.656503 2567 scope.go:117] "RemoveContainer" containerID="e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5" May 26 16:17:00.663327 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.663312 2567 scope.go:117] "RemoveContainer" containerID="98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd" May 26 16:17:00.667182 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.667160 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj"] May 26 16:17:00.670718 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.670701 2567 scope.go:117] "RemoveContainer" containerID="5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf" May 26 16:17:00.671013 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:17:00.670989 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf\": container with ID starting with 5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf not found: ID does not exist" containerID="5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf" May 26 16:17:00.671078 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.671025 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf"} err="failed to get container status \"5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf\": rpc error: code = NotFound desc = could not find container \"5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf\": container with ID starting with 5d5779bb6b7336eb28a70fc01764c556fc1ca4ce8a194c788b9e69eab218bdaf not found: ID does not exist" May 26 16:17:00.671078 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.671050 2567 scope.go:117] "RemoveContainer" containerID="e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5" May 26 16:17:00.671078 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.671070 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-dk8qj"] May 26 16:17:00.671317 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:17:00.671298 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5\": container with ID starting with e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5 not found: ID does not exist" containerID="e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5" May 26 16:17:00.671363 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.671322 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5"} err="failed to get container status \"e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5\": rpc error: code = NotFound desc = could not find container \"e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5\": container with ID starting with e57fc5fc659cbc79edcb12fcc792f3437c2f7a95a29f092ce18741e920eae6a5 not found: ID does not exist" May 26 16:17:00.671363 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.671339 2567 scope.go:117] "RemoveContainer" containerID="98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd" May 26 16:17:00.671562 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:17:00.671547 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd\": container with ID starting with 98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd not found: ID does not exist" containerID="98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd" May 26 16:17:00.671603 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:00.671568 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd"} err="failed to get container status \"98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd\": rpc error: code = NotFound desc = could not find container \"98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd\": container with ID starting with 98631f3bcdb3aa897d246bd5c81154528803f499d0e8da638e62267a58b827fd not found: ID does not exist" May 26 16:17:02.049806 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:02.049775 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9476710-d812-4e7d-ae40-bebf65649165" path="/var/lib/kubelet/pods/f9476710-d812-4e7d-ae40-bebf65649165/volumes" May 26 16:17:03.641134 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:03.641108 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:17:03.641667 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:03.641644 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" May 26 16:17:04.526750 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:04.526723 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:17:13.642029 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:13.641920 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" May 26 16:17:23.642143 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:23.642100 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" May 26 16:17:33.642551 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:33.642510 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" May 26 16:17:43.642104 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:17:43.642075 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:20:48.040213 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:20:48.040186 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:20:48.041915 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:20:48.041896 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:25:31.199500 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.199468 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8"] May 26 16:25:31.201880 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.199870 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" containerID="cri-o://e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea" gracePeriod=30 May 26 16:25:31.201880 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.199945 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kube-rbac-proxy" containerID="cri-o://a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f" gracePeriod=30 May 26 16:25:31.289325 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.289289 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25"] May 26 16:25:31.289566 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.289554 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kube-rbac-proxy" May 26 16:25:31.289608 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.289568 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kube-rbac-proxy" May 26 16:25:31.289608 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.289582 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="storage-initializer" May 26 16:25:31.289608 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.289588 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="storage-initializer" May 26 16:25:31.289608 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.289598 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" May 26 16:25:31.289608 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.289603 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" May 26 16:25:31.289757 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.289649 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kserve-container" May 26 16:25:31.289757 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.289656 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9476710-d812-4e7d-ae40-bebf65649165" containerName="kube-rbac-proxy" May 26 16:25:31.292446 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.292426 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.294595 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.294569 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-007a5-kube-rbac-proxy-sar-config\"" May 26 16:25:31.294595 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.294590 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-007a5-predictor-serving-cert\"" May 26 16:25:31.303332 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.303300 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25"] May 26 16:25:31.420369 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.420332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c00405-6776-4b04-8d3a-7dc4d309da35-error-404-isvc-007a5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.420540 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.420397 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krh8q\" (UniqueName: \"kubernetes.io/projected/b0c00405-6776-4b04-8d3a-7dc4d309da35-kube-api-access-krh8q\") pod \"error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.420540 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.420423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c00405-6776-4b04-8d3a-7dc4d309da35-proxy-tls\") pod \"error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.521511 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.521424 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-krh8q\" (UniqueName: \"kubernetes.io/projected/b0c00405-6776-4b04-8d3a-7dc4d309da35-kube-api-access-krh8q\") pod \"error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.521511 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.521470 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c00405-6776-4b04-8d3a-7dc4d309da35-proxy-tls\") pod \"error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.521511 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.521506 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c00405-6776-4b04-8d3a-7dc4d309da35-error-404-isvc-007a5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.522166 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.522144 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c00405-6776-4b04-8d3a-7dc4d309da35-error-404-isvc-007a5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.523973 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.523956 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c00405-6776-4b04-8d3a-7dc4d309da35-proxy-tls\") pod \"error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.530304 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.530275 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krh8q\" (UniqueName: \"kubernetes.io/projected/b0c00405-6776-4b04-8d3a-7dc4d309da35-kube-api-access-krh8q\") pod \"error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.604105 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.604076 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:31.726595 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.726414 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25"] May 26 16:25:31.730167 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:25:31.730136 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c00405_6776_4b04_8d3a_7dc4d309da35.slice/crio-b4e14d16146510a06a8dcd7fc0e1379d928be7e0883986b3c163bfc7d98bd9d0 WatchSource:0}: Error finding container b4e14d16146510a06a8dcd7fc0e1379d928be7e0883986b3c163bfc7d98bd9d0: Status 404 returned error can't find the container with id b4e14d16146510a06a8dcd7fc0e1379d928be7e0883986b3c163bfc7d98bd9d0 May 26 16:25:31.732241 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:31.732222 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:25:32.007836 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:32.007790 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" event={"ID":"b0c00405-6776-4b04-8d3a-7dc4d309da35","Type":"ContainerStarted","Data":"0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f"} May 26 16:25:32.007836 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:32.007840 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" event={"ID":"b0c00405-6776-4b04-8d3a-7dc4d309da35","Type":"ContainerStarted","Data":"d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099"} May 26 16:25:32.008101 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:32.007857 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" event={"ID":"b0c00405-6776-4b04-8d3a-7dc4d309da35","Type":"ContainerStarted","Data":"b4e14d16146510a06a8dcd7fc0e1379d928be7e0883986b3c163bfc7d98bd9d0"} May 26 16:25:32.008101 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:32.007892 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:32.009282 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:32.009259 2567 generic.go:358] "Generic (PLEG): container finished" podID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerID="a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f" exitCode=2 May 26 16:25:32.009389 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:32.009294 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" event={"ID":"2fd1085c-0886-4d4d-b4b6-eeca8be643eb","Type":"ContainerDied","Data":"a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f"} May 26 16:25:32.024568 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:32.024518 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podStartSLOduration=1.024504283 podStartE2EDuration="1.024504283s" podCreationTimestamp="2026-05-26 16:25:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:25:32.02372374 +0000 UTC m=+1484.586456694" watchObservedRunningTime="2026-05-26 16:25:32.024504283 +0000 UTC m=+1484.587237338" May 26 16:25:33.012152 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:33.012114 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:33.013303 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:33.013277 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" May 26 16:25:34.014993 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.014954 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" May 26 16:25:34.445616 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.445594 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:25:34.546604 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.546521 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb4gr\" (UniqueName: \"kubernetes.io/projected/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-kube-api-access-pb4gr\") pod \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " May 26 16:25:34.546604 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.546574 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-error-404-isvc-df0d2-kube-rbac-proxy-sar-config\") pod \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " May 26 16:25:34.546773 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.546629 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-proxy-tls\") pod \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\" (UID: \"2fd1085c-0886-4d4d-b4b6-eeca8be643eb\") " May 26 16:25:34.546973 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.546929 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-error-404-isvc-df0d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-df0d2-kube-rbac-proxy-sar-config") pod "2fd1085c-0886-4d4d-b4b6-eeca8be643eb" (UID: "2fd1085c-0886-4d4d-b4b6-eeca8be643eb"). InnerVolumeSpecName "error-404-isvc-df0d2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:25:34.548619 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.548598 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-kube-api-access-pb4gr" (OuterVolumeSpecName: "kube-api-access-pb4gr") pod "2fd1085c-0886-4d4d-b4b6-eeca8be643eb" (UID: "2fd1085c-0886-4d4d-b4b6-eeca8be643eb"). InnerVolumeSpecName "kube-api-access-pb4gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:25:34.548812 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.548787 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2fd1085c-0886-4d4d-b4b6-eeca8be643eb" (UID: "2fd1085c-0886-4d4d-b4b6-eeca8be643eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:25:34.647570 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.647528 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4gr\" (UniqueName: \"kubernetes.io/projected/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-kube-api-access-pb4gr\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:25:34.647570 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.647566 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-error-404-isvc-df0d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:25:34.647756 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:34.647582 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd1085c-0886-4d4d-b4b6-eeca8be643eb-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:25:35.018684 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.018646 2567 generic.go:358] "Generic (PLEG): container finished" podID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerID="e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea" exitCode=0 May 26 16:25:35.019175 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.018734 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" May 26 16:25:35.019175 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.018736 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" event={"ID":"2fd1085c-0886-4d4d-b4b6-eeca8be643eb","Type":"ContainerDied","Data":"e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea"} May 26 16:25:35.019175 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.018789 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8" event={"ID":"2fd1085c-0886-4d4d-b4b6-eeca8be643eb","Type":"ContainerDied","Data":"1d721289a65bfb407e55b89db4189f7657abf57b4e200f778cf9820c6884fbe3"} May 26 16:25:35.019175 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.018819 2567 scope.go:117] "RemoveContainer" containerID="a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f" May 26 16:25:35.027087 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.027062 2567 scope.go:117] "RemoveContainer" containerID="e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea" May 26 16:25:35.033718 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.033702 2567 scope.go:117] "RemoveContainer" containerID="a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f" May 26 16:25:35.033982 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:25:35.033959 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f\": container with ID starting with a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f not found: ID does not exist" containerID="a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f" May 26 16:25:35.034076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.033991 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f"} err="failed to get container status \"a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f\": rpc error: code = NotFound desc = could not find container \"a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f\": container with ID starting with a86a0007fe3a4518afe628932e0b235092e1868d592f43dba63ecc04ce2a481f not found: ID does not exist" May 26 16:25:35.034076 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.034011 2567 scope.go:117] "RemoveContainer" containerID="e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea" May 26 16:25:35.034223 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:25:35.034205 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea\": container with ID starting with e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea not found: ID does not exist" containerID="e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea" May 26 16:25:35.034259 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.034230 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea"} err="failed to get container status \"e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea\": rpc error: code = NotFound desc = could not find container \"e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea\": container with ID starting with e33ac8b27f179c790d9bdd53e41943104b8019c881d622566a841fe6b7b807ea not found: ID does not exist" May 26 16:25:35.042764 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.042745 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8"] May 26 16:25:35.046970 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:35.046949 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df0d2-predictor-66496f47bf-q49z8"] May 26 16:25:36.049568 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:36.049534 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" path="/var/lib/kubelet/pods/2fd1085c-0886-4d4d-b4b6-eeca8be643eb/volumes" May 26 16:25:39.019966 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:39.019921 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:25:39.020362 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:39.020338 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" May 26 16:25:48.062384 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:48.062354 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:25:48.064289 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:48.064274 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:25:49.021048 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:49.021003 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" May 26 16:25:59.020698 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:25:59.020656 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" May 26 16:26:09.020871 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:09.020786 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" May 26 16:26:10.999265 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:10.999232 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd"] May 26 16:26:10.999664 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:10.999516 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" containerID="cri-o://772af26713786e488fed44a203bc3c36e6c3407b20f6e813cf774c4953534969" gracePeriod=30 May 26 16:26:10.999664 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:10.999565 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kube-rbac-proxy" containerID="cri-o://498150941b24224b73570fac55b238c05d28ab7fa66e6aff4dc632f1312e03de" gracePeriod=30 May 26 16:26:11.039913 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.039881 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx"] May 26 16:26:11.040178 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.040166 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" May 26 16:26:11.040233 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.040180 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" May 26 16:26:11.040233 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.040188 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kube-rbac-proxy" May 26 16:26:11.040233 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.040194 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kube-rbac-proxy" May 26 16:26:11.040233 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.040238 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kube-rbac-proxy" May 26 16:26:11.040379 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.040246 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="2fd1085c-0886-4d4d-b4b6-eeca8be643eb" containerName="kserve-container" May 26 16:26:11.044265 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.044249 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.046609 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.046583 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a007a-kube-rbac-proxy-sar-config\"" May 26 16:26:11.046728 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.046615 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a007a-predictor-serving-cert\"" May 26 16:26:11.052871 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.052848 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx"] May 26 16:26:11.120449 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.120421 2567 generic.go:358] "Generic (PLEG): container finished" podID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerID="498150941b24224b73570fac55b238c05d28ab7fa66e6aff4dc632f1312e03de" exitCode=2 May 26 16:26:11.120580 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.120503 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" event={"ID":"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0","Type":"ContainerDied","Data":"498150941b24224b73570fac55b238c05d28ab7fa66e6aff4dc632f1312e03de"} May 26 16:26:11.132874 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.132847 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8430529d-fb3d-44c3-836d-8f123b3dc2a6-error-404-isvc-a007a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.133006 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.132892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8430529d-fb3d-44c3-836d-8f123b3dc2a6-proxy-tls\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.133006 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.132925 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5wz\" (UniqueName: \"kubernetes.io/projected/8430529d-fb3d-44c3-836d-8f123b3dc2a6-kube-api-access-sg5wz\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.233875 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.233833 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8430529d-fb3d-44c3-836d-8f123b3dc2a6-error-404-isvc-a007a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.233875 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.233881 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8430529d-fb3d-44c3-836d-8f123b3dc2a6-proxy-tls\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.234143 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.233901 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5wz\" (UniqueName: \"kubernetes.io/projected/8430529d-fb3d-44c3-836d-8f123b3dc2a6-kube-api-access-sg5wz\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.234143 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:26:11.234025 2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a007a-predictor-serving-cert: secret "error-404-isvc-a007a-predictor-serving-cert" not found May 26 16:26:11.234143 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:26:11.234114 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8430529d-fb3d-44c3-836d-8f123b3dc2a6-proxy-tls podName:8430529d-fb3d-44c3-836d-8f123b3dc2a6 nodeName:}" failed. No retries permitted until 2026-05-26 16:26:11.734096042 +0000 UTC m=+1524.296828963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8430529d-fb3d-44c3-836d-8f123b3dc2a6-proxy-tls") pod "error-404-isvc-a007a-predictor-c5d584875-l64qx" (UID: "8430529d-fb3d-44c3-836d-8f123b3dc2a6") : secret "error-404-isvc-a007a-predictor-serving-cert" not found May 26 16:26:11.234598 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.234570 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8430529d-fb3d-44c3-836d-8f123b3dc2a6-error-404-isvc-a007a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.242660 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.242631 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5wz\" (UniqueName: \"kubernetes.io/projected/8430529d-fb3d-44c3-836d-8f123b3dc2a6-kube-api-access-sg5wz\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.737245 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.737191 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8430529d-fb3d-44c3-836d-8f123b3dc2a6-proxy-tls\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.739580 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.739548 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8430529d-fb3d-44c3-836d-8f123b3dc2a6-proxy-tls\") pod \"error-404-isvc-a007a-predictor-c5d584875-l64qx\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:11.955394 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:11.955356 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:12.080617 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:12.080586 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx"] May 26 16:26:12.083422 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:26:12.083387 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8430529d_fb3d_44c3_836d_8f123b3dc2a6.slice/crio-2ef2c58ed0ec5a0d3a2b97a52adce9e83ff9952be55a854c842bdd83845364b0 WatchSource:0}: Error finding container 2ef2c58ed0ec5a0d3a2b97a52adce9e83ff9952be55a854c842bdd83845364b0: Status 404 returned error can't find the container with id 2ef2c58ed0ec5a0d3a2b97a52adce9e83ff9952be55a854c842bdd83845364b0 May 26 16:26:12.124972 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:12.124923 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" event={"ID":"8430529d-fb3d-44c3-836d-8f123b3dc2a6","Type":"ContainerStarted","Data":"2ef2c58ed0ec5a0d3a2b97a52adce9e83ff9952be55a854c842bdd83845364b0"} May 26 16:26:13.129211 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:13.129175 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" event={"ID":"8430529d-fb3d-44c3-836d-8f123b3dc2a6","Type":"ContainerStarted","Data":"717ab7690e603d480d56d7348acafa96f71d358881e7d514194207109cf60417"} May 26 16:26:13.129211 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:13.129211 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" event={"ID":"8430529d-fb3d-44c3-836d-8f123b3dc2a6","Type":"ContainerStarted","Data":"71958ba920e44e01f1540c6986516801d38d469a5cb4816f55edc93632007411"} May 26 16:26:13.129628 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:13.129331 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:13.147292 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:13.147249 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" podStartSLOduration=2.147236112 podStartE2EDuration="2.147236112s" podCreationTimestamp="2026-05-26 16:26:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:26:13.14588049 +0000 UTC m=+1525.708613437" watchObservedRunningTime="2026-05-26 16:26:13.147236112 +0000 UTC m=+1525.709969055" May 26 16:26:13.638288 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:13.638249 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused" May 26 16:26:13.642523 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:13.642501 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" May 26 16:26:14.133683 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.133656 2567 generic.go:358] "Generic (PLEG): container finished" podID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerID="772af26713786e488fed44a203bc3c36e6c3407b20f6e813cf774c4953534969" exitCode=0 May 26 16:26:14.134060 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.133735 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" event={"ID":"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0","Type":"ContainerDied","Data":"772af26713786e488fed44a203bc3c36e6c3407b20f6e813cf774c4953534969"} May 26 16:26:14.134129 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.134095 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:14.135209 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.135187 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" May 26 16:26:14.246034 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.246011 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:26:14.254109 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.254089 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-error-404-isvc-ad62d-kube-rbac-proxy-sar-config\") pod \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " May 26 16:26:14.254186 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.254158 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-proxy-tls\") pod \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " May 26 16:26:14.254186 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.254176 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h72jk\" (UniqueName: \"kubernetes.io/projected/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-kube-api-access-h72jk\") pod \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\" (UID: \"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0\") " May 26 16:26:14.254459 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.254433 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-error-404-isvc-ad62d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-ad62d-kube-rbac-proxy-sar-config") pod "e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" (UID: "e8cc1d7e-b614-43f6-acbf-7d42248fbbe0"). InnerVolumeSpecName "error-404-isvc-ad62d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:26:14.256135 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.256109 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" (UID: "e8cc1d7e-b614-43f6-acbf-7d42248fbbe0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:26:14.256360 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.256338 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-kube-api-access-h72jk" (OuterVolumeSpecName: "kube-api-access-h72jk") pod "e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" (UID: "e8cc1d7e-b614-43f6-acbf-7d42248fbbe0"). InnerVolumeSpecName "kube-api-access-h72jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:26:14.355557 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.355521 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:26:14.355557 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.355550 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h72jk\" (UniqueName: \"kubernetes.io/projected/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-kube-api-access-h72jk\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:26:14.355557 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:14.355561 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0-error-404-isvc-ad62d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:26:15.137711 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:15.137672 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" event={"ID":"e8cc1d7e-b614-43f6-acbf-7d42248fbbe0","Type":"ContainerDied","Data":"0e79c19a393f8f6f619816e88a8a04c798780909486c91a8f5c4e2ded50fa9a1"} May 26 16:26:15.137711 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:15.137720 2567 scope.go:117] "RemoveContainer" containerID="498150941b24224b73570fac55b238c05d28ab7fa66e6aff4dc632f1312e03de" May 26 16:26:15.138249 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:15.137725 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd" May 26 16:26:15.138249 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:15.138023 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" May 26 16:26:15.148582 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:15.148560 2567 scope.go:117] "RemoveContainer" containerID="772af26713786e488fed44a203bc3c36e6c3407b20f6e813cf774c4953534969" May 26 16:26:15.162106 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:15.162079 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd"] May 26 16:26:15.166424 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:15.166390 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad62d-predictor-69787655ff-l67xd"] May 26 16:26:16.048872 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:16.048841 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" path="/var/lib/kubelet/pods/e8cc1d7e-b614-43f6-acbf-7d42248fbbe0/volumes" May 26 16:26:19.021828 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:19.021799 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:26:20.143281 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:20.143253 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:26:20.143777 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:20.143752 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" May 26 16:26:30.143916 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:30.143878 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" May 26 16:26:40.144049 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:40.144011 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" May 26 16:26:41.317249 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.317169 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25"] May 26 16:26:41.317623 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.317469 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" containerID="cri-o://d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099" gracePeriod=30 May 26 16:26:41.317623 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.317521 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kube-rbac-proxy" containerID="cri-o://0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f" gracePeriod=30 May 26 16:26:41.498813 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.498773 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts"] May 26 16:26:41.499181 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.499164 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kube-rbac-proxy" May 26 16:26:41.499277 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.499184 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kube-rbac-proxy" May 26 16:26:41.499277 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.499201 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" May 26 16:26:41.499277 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.499209 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" May 26 16:26:41.499277 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.499272 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kube-rbac-proxy" May 26 16:26:41.499491 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.499286 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8cc1d7e-b614-43f6-acbf-7d42248fbbe0" containerName="kserve-container" May 26 16:26:41.502284 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.502263 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:41.504882 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.504856 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2138e-predictor-serving-cert\"" May 26 16:26:41.505011 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.504921 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2138e-kube-rbac-proxy-sar-config\"" May 26 16:26:41.512736 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.512707 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts"] May 26 16:26:41.557113 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.557079 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7339f829-8832-48c6-ab59-28c69db0f320-error-404-isvc-2138e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:41.557113 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.557115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x96h\" (UniqueName: \"kubernetes.io/projected/7339f829-8832-48c6-ab59-28c69db0f320-kube-api-access-6x96h\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:41.557285 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.557140 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7339f829-8832-48c6-ab59-28c69db0f320-proxy-tls\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:41.658511 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.658478 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7339f829-8832-48c6-ab59-28c69db0f320-error-404-isvc-2138e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:41.658511 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.658513 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6x96h\" (UniqueName: \"kubernetes.io/projected/7339f829-8832-48c6-ab59-28c69db0f320-kube-api-access-6x96h\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:41.658771 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.658534 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7339f829-8832-48c6-ab59-28c69db0f320-proxy-tls\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:41.658771 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:26:41.658647 2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2138e-predictor-serving-cert: secret "error-404-isvc-2138e-predictor-serving-cert" not found May 26 16:26:41.658771 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:26:41.658718 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7339f829-8832-48c6-ab59-28c69db0f320-proxy-tls podName:7339f829-8832-48c6-ab59-28c69db0f320 nodeName:}" failed. No retries permitted until 2026-05-26 16:26:42.158697021 +0000 UTC m=+1554.721429943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7339f829-8832-48c6-ab59-28c69db0f320-proxy-tls") pod "error-404-isvc-2138e-predictor-f5c8b557f-z8xts" (UID: "7339f829-8832-48c6-ab59-28c69db0f320") : secret "error-404-isvc-2138e-predictor-serving-cert" not found May 26 16:26:41.659242 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.659214 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7339f829-8832-48c6-ab59-28c69db0f320-error-404-isvc-2138e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:41.667209 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:41.667183 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x96h\" (UniqueName: \"kubernetes.io/projected/7339f829-8832-48c6-ab59-28c69db0f320-kube-api-access-6x96h\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:42.162268 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:42.162236 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7339f829-8832-48c6-ab59-28c69db0f320-proxy-tls\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:42.164639 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:42.164611 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7339f829-8832-48c6-ab59-28c69db0f320-proxy-tls\") pod \"error-404-isvc-2138e-predictor-f5c8b557f-z8xts\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:42.209802 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:42.209766 2567 generic.go:358] "Generic (PLEG): container finished" podID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerID="0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f" exitCode=2 May 26 16:26:42.209977 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:42.209824 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" event={"ID":"b0c00405-6776-4b04-8d3a-7dc4d309da35","Type":"ContainerDied","Data":"0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f"} May 26 16:26:42.412444 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:42.412355 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:42.548259 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:42.548235 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts"] May 26 16:26:42.550117 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:26:42.550086 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7339f829_8832_48c6_ab59_28c69db0f320.slice/crio-41d8a513deef5e8e00193620e1dd9fc71c0a2f2ed62e8ed35d642ffdde96b03e WatchSource:0}: Error finding container 41d8a513deef5e8e00193620e1dd9fc71c0a2f2ed62e8ed35d642ffdde96b03e: Status 404 returned error can't find the container with id 41d8a513deef5e8e00193620e1dd9fc71c0a2f2ed62e8ed35d642ffdde96b03e May 26 16:26:43.214204 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:43.214173 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" event={"ID":"7339f829-8832-48c6-ab59-28c69db0f320","Type":"ContainerStarted","Data":"0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871"} May 26 16:26:43.214204 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:43.214206 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" event={"ID":"7339f829-8832-48c6-ab59-28c69db0f320","Type":"ContainerStarted","Data":"d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90"} May 26 16:26:43.214461 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:43.214216 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" event={"ID":"7339f829-8832-48c6-ab59-28c69db0f320","Type":"ContainerStarted","Data":"41d8a513deef5e8e00193620e1dd9fc71c0a2f2ed62e8ed35d642ffdde96b03e"} May 26 16:26:43.214461 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:43.214364 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:43.232039 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:43.231996 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" podStartSLOduration=2.231982444 podStartE2EDuration="2.231982444s" podCreationTimestamp="2026-05-26 16:26:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:26:43.230729586 +0000 UTC m=+1555.793462543" watchObservedRunningTime="2026-05-26 16:26:43.231982444 +0000 UTC m=+1555.794715387" May 26 16:26:44.015724 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.015677 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused" May 26 16:26:44.217322 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.217288 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:44.218474 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.218450 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" May 26 16:26:44.473412 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.473385 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:26:44.584392 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.584315 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krh8q\" (UniqueName: \"kubernetes.io/projected/b0c00405-6776-4b04-8d3a-7dc4d309da35-kube-api-access-krh8q\") pod \"b0c00405-6776-4b04-8d3a-7dc4d309da35\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " May 26 16:26:44.584392 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.584362 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c00405-6776-4b04-8d3a-7dc4d309da35-proxy-tls\") pod \"b0c00405-6776-4b04-8d3a-7dc4d309da35\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " May 26 16:26:44.584392 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.584392 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c00405-6776-4b04-8d3a-7dc4d309da35-error-404-isvc-007a5-kube-rbac-proxy-sar-config\") pod \"b0c00405-6776-4b04-8d3a-7dc4d309da35\" (UID: \"b0c00405-6776-4b04-8d3a-7dc4d309da35\") " May 26 16:26:44.584737 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.584712 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c00405-6776-4b04-8d3a-7dc4d309da35-error-404-isvc-007a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-007a5-kube-rbac-proxy-sar-config") pod "b0c00405-6776-4b04-8d3a-7dc4d309da35" (UID: "b0c00405-6776-4b04-8d3a-7dc4d309da35"). InnerVolumeSpecName "error-404-isvc-007a5-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:26:44.586410 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.586391 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c00405-6776-4b04-8d3a-7dc4d309da35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0c00405-6776-4b04-8d3a-7dc4d309da35" (UID: "b0c00405-6776-4b04-8d3a-7dc4d309da35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:26:44.586507 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.586492 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c00405-6776-4b04-8d3a-7dc4d309da35-kube-api-access-krh8q" (OuterVolumeSpecName: "kube-api-access-krh8q") pod "b0c00405-6776-4b04-8d3a-7dc4d309da35" (UID: "b0c00405-6776-4b04-8d3a-7dc4d309da35"). InnerVolumeSpecName "kube-api-access-krh8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:26:44.685621 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.685586 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krh8q\" (UniqueName: \"kubernetes.io/projected/b0c00405-6776-4b04-8d3a-7dc4d309da35-kube-api-access-krh8q\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:26:44.685621 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.685615 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c00405-6776-4b04-8d3a-7dc4d309da35-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:26:44.685621 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:44.685628 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c00405-6776-4b04-8d3a-7dc4d309da35-error-404-isvc-007a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:26:45.220765 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.220729 2567 generic.go:358] "Generic (PLEG): container finished" podID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerID="d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099" exitCode=0 May 26 16:26:45.221179 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.220814 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" May 26 16:26:45.221179 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.220814 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" event={"ID":"b0c00405-6776-4b04-8d3a-7dc4d309da35","Type":"ContainerDied","Data":"d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099"} May 26 16:26:45.221179 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.220851 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25" event={"ID":"b0c00405-6776-4b04-8d3a-7dc4d309da35","Type":"ContainerDied","Data":"b4e14d16146510a06a8dcd7fc0e1379d928be7e0883986b3c163bfc7d98bd9d0"} May 26 16:26:45.221179 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.220866 2567 scope.go:117] "RemoveContainer" containerID="0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f" May 26 16:26:45.221360 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.221338 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" May 26 16:26:45.229160 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.229142 2567 scope.go:117] "RemoveContainer" containerID="d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099" May 26 16:26:45.235910 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.235893 2567 scope.go:117] "RemoveContainer" containerID="0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f" May 26 16:26:45.236139 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:26:45.236124 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f\": container with ID starting with 0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f not found: ID does not exist" containerID="0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f" May 26 16:26:45.236182 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.236148 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f"} err="failed to get container status \"0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f\": rpc error: code = NotFound desc = could not find container \"0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f\": container with ID starting with 0835f07ed4bd2736ee58a60054edc3c3587a8f11484f3c301c658dc41820548f not found: ID does not exist" May 26 16:26:45.236182 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.236172 2567 scope.go:117] "RemoveContainer" containerID="d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099" May 26 16:26:45.236372 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:26:45.236357 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099\": container with ID starting with d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099 not found: ID does not exist" containerID="d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099" May 26 16:26:45.236425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.236375 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099"} err="failed to get container status \"d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099\": rpc error: code = NotFound desc = could not find container \"d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099\": container with ID starting with d8f3b22ca7887ac720db7c7d5f8a6a87974c1325c419b003f243b895fa427099 not found: ID does not exist" May 26 16:26:45.242118 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.242098 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25"] May 26 16:26:45.246033 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:45.246010 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-007a5-predictor-d4b6c4bd7-9hz25"] May 26 16:26:46.049761 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:46.049728 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" path="/var/lib/kubelet/pods/b0c00405-6776-4b04-8d3a-7dc4d309da35/volumes" May 26 16:26:50.144129 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:50.144089 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" May 26 16:26:50.225860 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:50.225832 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:26:50.226255 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:26:50.226231 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" May 26 16:27:00.144747 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:00.144719 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:27:00.226665 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:00.226633 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" May 26 16:27:10.226541 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:10.226503 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" May 26 16:27:20.226979 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:20.226919 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" May 26 16:27:21.209040 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.209002 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx"] May 26 16:27:21.209332 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.209285 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" containerID="cri-o://71958ba920e44e01f1540c6986516801d38d469a5cb4816f55edc93632007411" gracePeriod=30 May 26 16:27:21.209509 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.209333 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kube-rbac-proxy" containerID="cri-o://717ab7690e603d480d56d7348acafa96f71d358881e7d514194207109cf60417" gracePeriod=30 May 26 16:27:21.268058 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.268020 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x"] May 26 16:27:21.268491 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.268447 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" May 26 16:27:21.268491 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.268465 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" May 26 16:27:21.268491 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.268480 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kube-rbac-proxy" May 26 16:27:21.268491 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.268487 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kube-rbac-proxy" May 26 16:27:21.268690 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.268567 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kserve-container" May 26 16:27:21.268690 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.268579 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0c00405-6776-4b04-8d3a-7dc4d309da35" containerName="kube-rbac-proxy" May 26 16:27:21.271575 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.271554 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.273839 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.273814 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-86373-predictor-serving-cert\"" May 26 16:27:21.273975 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.273816 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-86373-kube-rbac-proxy-sar-config\"" May 26 16:27:21.274559 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.274540 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0269a7-24fc-4ac2-8813-52425891e08e-error-404-isvc-86373-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.274615 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.274596 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0269a7-24fc-4ac2-8813-52425891e08e-proxy-tls\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.274672 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.274623 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5f7d\" (UniqueName: \"kubernetes.io/projected/9f0269a7-24fc-4ac2-8813-52425891e08e-kube-api-access-t5f7d\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.281593 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.281568 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x"] May 26 16:27:21.375105 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.375066 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0269a7-24fc-4ac2-8813-52425891e08e-error-404-isvc-86373-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.375274 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.375119 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0269a7-24fc-4ac2-8813-52425891e08e-proxy-tls\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.375274 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.375152 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t5f7d\" (UniqueName: \"kubernetes.io/projected/9f0269a7-24fc-4ac2-8813-52425891e08e-kube-api-access-t5f7d\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.375346 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:27:21.375262 2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-86373-predictor-serving-cert: secret "error-404-isvc-86373-predictor-serving-cert" not found May 26 16:27:21.375346 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:27:21.375335 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f0269a7-24fc-4ac2-8813-52425891e08e-proxy-tls podName:9f0269a7-24fc-4ac2-8813-52425891e08e nodeName:}" failed. No retries permitted until 2026-05-26 16:27:21.875313897 +0000 UTC m=+1594.438046836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f0269a7-24fc-4ac2-8813-52425891e08e-proxy-tls") pod "error-404-isvc-86373-predictor-776c7fd767-prf4x" (UID: "9f0269a7-24fc-4ac2-8813-52425891e08e") : secret "error-404-isvc-86373-predictor-serving-cert" not found May 26 16:27:21.375748 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.375727 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0269a7-24fc-4ac2-8813-52425891e08e-error-404-isvc-86373-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.383969 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.383949 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5f7d\" (UniqueName: \"kubernetes.io/projected/9f0269a7-24fc-4ac2-8813-52425891e08e-kube-api-access-t5f7d\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.878126 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.878080 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0269a7-24fc-4ac2-8813-52425891e08e-proxy-tls\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.880745 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.880714 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0269a7-24fc-4ac2-8813-52425891e08e-proxy-tls\") pod \"error-404-isvc-86373-predictor-776c7fd767-prf4x\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:21.883060 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:21.883038 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:22.005614 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:22.005571 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x"] May 26 16:27:22.009905 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:27:22.009877 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0269a7_24fc_4ac2_8813_52425891e08e.slice/crio-58fb5b618f341f5b25c54cf490ca8e2b4a42fff3b4e211c48d206c5b6d24c906 WatchSource:0}: Error finding container 58fb5b618f341f5b25c54cf490ca8e2b4a42fff3b4e211c48d206c5b6d24c906: Status 404 returned error can't find the container with id 58fb5b618f341f5b25c54cf490ca8e2b4a42fff3b4e211c48d206c5b6d24c906 May 26 16:27:22.321705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:22.321609 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" event={"ID":"9f0269a7-24fc-4ac2-8813-52425891e08e","Type":"ContainerStarted","Data":"2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe"} May 26 16:27:22.321705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:22.321656 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" event={"ID":"9f0269a7-24fc-4ac2-8813-52425891e08e","Type":"ContainerStarted","Data":"1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81"} May 26 16:27:22.321705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:22.321671 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" event={"ID":"9f0269a7-24fc-4ac2-8813-52425891e08e","Type":"ContainerStarted","Data":"58fb5b618f341f5b25c54cf490ca8e2b4a42fff3b4e211c48d206c5b6d24c906"} May 26 16:27:22.322197 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:22.321749 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:22.323130 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:22.323108 2567 generic.go:358] "Generic (PLEG): container finished" podID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerID="717ab7690e603d480d56d7348acafa96f71d358881e7d514194207109cf60417" exitCode=2 May 26 16:27:22.323215 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:22.323166 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" event={"ID":"8430529d-fb3d-44c3-836d-8f123b3dc2a6","Type":"ContainerDied","Data":"717ab7690e603d480d56d7348acafa96f71d358881e7d514194207109cf60417"} May 26 16:27:22.338799 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:22.338759 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podStartSLOduration=1.338747514 podStartE2EDuration="1.338747514s" podCreationTimestamp="2026-05-26 16:27:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:27:22.337552481 +0000 UTC m=+1594.900285448" watchObservedRunningTime="2026-05-26 16:27:22.338747514 +0000 UTC m=+1594.901480456" May 26 16:27:23.325686 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:23.325653 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:23.326864 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:23.326835 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" May 26 16:27:24.330121 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.330080 2567 generic.go:358] "Generic (PLEG): container finished" podID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerID="71958ba920e44e01f1540c6986516801d38d469a5cb4816f55edc93632007411" exitCode=0 May 26 16:27:24.330493 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.330149 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" event={"ID":"8430529d-fb3d-44c3-836d-8f123b3dc2a6","Type":"ContainerDied","Data":"71958ba920e44e01f1540c6986516801d38d469a5cb4816f55edc93632007411"} May 26 16:27:24.330493 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.330450 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" May 26 16:27:24.346479 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.346460 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:27:24.495538 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.495452 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8430529d-fb3d-44c3-836d-8f123b3dc2a6-error-404-isvc-a007a-kube-rbac-proxy-sar-config\") pod \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " May 26 16:27:24.495538 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.495506 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg5wz\" (UniqueName: \"kubernetes.io/projected/8430529d-fb3d-44c3-836d-8f123b3dc2a6-kube-api-access-sg5wz\") pod \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " May 26 16:27:24.495538 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.495529 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8430529d-fb3d-44c3-836d-8f123b3dc2a6-proxy-tls\") pod \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\" (UID: \"8430529d-fb3d-44c3-836d-8f123b3dc2a6\") " May 26 16:27:24.495819 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.495803 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8430529d-fb3d-44c3-836d-8f123b3dc2a6-error-404-isvc-a007a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-a007a-kube-rbac-proxy-sar-config") pod "8430529d-fb3d-44c3-836d-8f123b3dc2a6" (UID: "8430529d-fb3d-44c3-836d-8f123b3dc2a6"). InnerVolumeSpecName "error-404-isvc-a007a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:27:24.497597 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.497575 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8430529d-fb3d-44c3-836d-8f123b3dc2a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8430529d-fb3d-44c3-836d-8f123b3dc2a6" (UID: "8430529d-fb3d-44c3-836d-8f123b3dc2a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:27:24.497697 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.497657 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8430529d-fb3d-44c3-836d-8f123b3dc2a6-kube-api-access-sg5wz" (OuterVolumeSpecName: "kube-api-access-sg5wz") pod "8430529d-fb3d-44c3-836d-8f123b3dc2a6" (UID: "8430529d-fb3d-44c3-836d-8f123b3dc2a6"). InnerVolumeSpecName "kube-api-access-sg5wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:27:24.596745 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.596706 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8430529d-fb3d-44c3-836d-8f123b3dc2a6-error-404-isvc-a007a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:27:24.596745 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.596739 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg5wz\" (UniqueName: \"kubernetes.io/projected/8430529d-fb3d-44c3-836d-8f123b3dc2a6-kube-api-access-sg5wz\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:27:24.596745 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:24.596754 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8430529d-fb3d-44c3-836d-8f123b3dc2a6-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:27:25.334497 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:25.334454 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" event={"ID":"8430529d-fb3d-44c3-836d-8f123b3dc2a6","Type":"ContainerDied","Data":"2ef2c58ed0ec5a0d3a2b97a52adce9e83ff9952be55a854c842bdd83845364b0"} May 26 16:27:25.334497 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:25.334481 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx" May 26 16:27:25.335020 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:25.334512 2567 scope.go:117] "RemoveContainer" containerID="717ab7690e603d480d56d7348acafa96f71d358881e7d514194207109cf60417" May 26 16:27:25.342306 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:25.342142 2567 scope.go:117] "RemoveContainer" containerID="71958ba920e44e01f1540c6986516801d38d469a5cb4816f55edc93632007411" May 26 16:27:25.357448 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:25.357426 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx"] May 26 16:27:25.361324 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:25.361303 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a007a-predictor-c5d584875-l64qx"] May 26 16:27:26.049851 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:26.049818 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" path="/var/lib/kubelet/pods/8430529d-fb3d-44c3-836d-8f123b3dc2a6/volumes" May 26 16:27:29.335223 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:29.335191 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:27:29.335726 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:29.335702 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" May 26 16:27:30.227098 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:30.227070 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:27:39.336751 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:39.336656 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" May 26 16:27:49.336263 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:49.336221 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" May 26 16:27:59.336031 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:27:59.335991 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" May 26 16:28:09.337251 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:28:09.337220 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:30:48.079618 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:30:48.079586 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:30:48.083371 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:30:48.083345 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:35:48.097670 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:48.097643 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:35:48.101581 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:48.101562 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:35:56.342039 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.341999 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts"] May 26 16:35:56.342499 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.342275 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" containerID="cri-o://d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90" gracePeriod=30 May 26 16:35:56.342499 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.342305 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kube-rbac-proxy" containerID="cri-o://0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871" gracePeriod=30 May 26 16:35:56.464030 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.464000 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp"] May 26 16:35:56.464296 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.464283 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" May 26 16:35:56.464338 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.464298 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" May 26 16:35:56.464338 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.464311 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kube-rbac-proxy" May 26 16:35:56.464338 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.464316 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kube-rbac-proxy" May 26 16:35:56.464426 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.464372 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kserve-container" May 26 16:35:56.464426 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.464378 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="8430529d-fb3d-44c3-836d-8f123b3dc2a6" containerName="kube-rbac-proxy" May 26 16:35:56.467024 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.467009 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.469263 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.469243 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-82fbc-predictor-serving-cert\"" May 26 16:35:56.469385 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.469246 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-82fbc-kube-rbac-proxy-sar-config\"" May 26 16:35:56.475602 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.475577 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp"] May 26 16:35:56.608427 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.608388 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75743ec9-c617-4958-a564-df9910f039a5-error-404-isvc-82fbc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-82fbc-predictor-855f648bc-wjldp\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.608596 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.608454 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq77m\" (UniqueName: \"kubernetes.io/projected/75743ec9-c617-4958-a564-df9910f039a5-kube-api-access-vq77m\") pod \"error-404-isvc-82fbc-predictor-855f648bc-wjldp\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.608596 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.608547 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75743ec9-c617-4958-a564-df9910f039a5-proxy-tls\") pod \"error-404-isvc-82fbc-predictor-855f648bc-wjldp\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.692177 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.692139 2567 generic.go:358] "Generic (PLEG): container finished" podID="7339f829-8832-48c6-ab59-28c69db0f320" containerID="0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871" exitCode=2 May 26 16:35:56.692329 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.692204 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" event={"ID":"7339f829-8832-48c6-ab59-28c69db0f320","Type":"ContainerDied","Data":"0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871"} May 26 16:35:56.709128 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.709101 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vq77m\" (UniqueName: \"kubernetes.io/projected/75743ec9-c617-4958-a564-df9910f039a5-kube-api-access-vq77m\") pod \"error-404-isvc-82fbc-predictor-855f648bc-wjldp\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.709274 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.709142 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75743ec9-c617-4958-a564-df9910f039a5-proxy-tls\") pod \"error-404-isvc-82fbc-predictor-855f648bc-wjldp\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.709274 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.709177 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75743ec9-c617-4958-a564-df9910f039a5-error-404-isvc-82fbc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-82fbc-predictor-855f648bc-wjldp\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.709739 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.709716 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75743ec9-c617-4958-a564-df9910f039a5-error-404-isvc-82fbc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-82fbc-predictor-855f648bc-wjldp\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.711623 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.711602 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75743ec9-c617-4958-a564-df9910f039a5-proxy-tls\") pod \"error-404-isvc-82fbc-predictor-855f648bc-wjldp\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.718645 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.718622 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq77m\" (UniqueName: \"kubernetes.io/projected/75743ec9-c617-4958-a564-df9910f039a5-kube-api-access-vq77m\") pod \"error-404-isvc-82fbc-predictor-855f648bc-wjldp\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.778112 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.778074 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:56.906695 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.906619 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp"] May 26 16:35:56.909326 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:35:56.909296 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75743ec9_c617_4958_a564_df9910f039a5.slice/crio-78cf6aea96cbb7e1a650152b8f2631fff3ec7ad8031bb0a3ca1c437410caeb62 WatchSource:0}: Error finding container 78cf6aea96cbb7e1a650152b8f2631fff3ec7ad8031bb0a3ca1c437410caeb62: Status 404 returned error can't find the container with id 78cf6aea96cbb7e1a650152b8f2631fff3ec7ad8031bb0a3ca1c437410caeb62 May 26 16:35:56.911010 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:56.910994 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:35:57.697494 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:57.697457 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" event={"ID":"75743ec9-c617-4958-a564-df9910f039a5","Type":"ContainerStarted","Data":"04365e47ee340172d2ea78accb76f1324cf396f07ae6a7aba4a39222aaa10ef0"} May 26 16:35:57.697494 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:57.697492 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" event={"ID":"75743ec9-c617-4958-a564-df9910f039a5","Type":"ContainerStarted","Data":"bd1b191ecdc88469428e5652dfa9e847b7e18063d2f4d1861949911561742cb8"} May 26 16:35:57.697494 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:57.697503 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" event={"ID":"75743ec9-c617-4958-a564-df9910f039a5","Type":"ContainerStarted","Data":"78cf6aea96cbb7e1a650152b8f2631fff3ec7ad8031bb0a3ca1c437410caeb62"} May 26 16:35:57.698020 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:57.697588 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:57.721382 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:57.721331 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podStartSLOduration=1.721315948 podStartE2EDuration="1.721315948s" podCreationTimestamp="2026-05-26 16:35:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:35:57.714882092 +0000 UTC m=+2110.277615035" watchObservedRunningTime="2026-05-26 16:35:57.721315948 +0000 UTC m=+2110.284048890" May 26 16:35:58.700137 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:58.700101 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:35:58.701409 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:58.701384 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" May 26 16:35:59.382237 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.382213 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:35:59.530316 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.530224 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7339f829-8832-48c6-ab59-28c69db0f320-proxy-tls\") pod \"7339f829-8832-48c6-ab59-28c69db0f320\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " May 26 16:35:59.530316 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.530302 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7339f829-8832-48c6-ab59-28c69db0f320-error-404-isvc-2138e-kube-rbac-proxy-sar-config\") pod \"7339f829-8832-48c6-ab59-28c69db0f320\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " May 26 16:35:59.530550 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.530330 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x96h\" (UniqueName: \"kubernetes.io/projected/7339f829-8832-48c6-ab59-28c69db0f320-kube-api-access-6x96h\") pod \"7339f829-8832-48c6-ab59-28c69db0f320\" (UID: \"7339f829-8832-48c6-ab59-28c69db0f320\") " May 26 16:35:59.530701 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.530678 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7339f829-8832-48c6-ab59-28c69db0f320-error-404-isvc-2138e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-2138e-kube-rbac-proxy-sar-config") pod "7339f829-8832-48c6-ab59-28c69db0f320" (UID: "7339f829-8832-48c6-ab59-28c69db0f320"). InnerVolumeSpecName "error-404-isvc-2138e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:35:59.532418 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.532394 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7339f829-8832-48c6-ab59-28c69db0f320-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7339f829-8832-48c6-ab59-28c69db0f320" (UID: "7339f829-8832-48c6-ab59-28c69db0f320"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:35:59.532543 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.532522 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7339f829-8832-48c6-ab59-28c69db0f320-kube-api-access-6x96h" (OuterVolumeSpecName: "kube-api-access-6x96h") pod "7339f829-8832-48c6-ab59-28c69db0f320" (UID: "7339f829-8832-48c6-ab59-28c69db0f320"). InnerVolumeSpecName "kube-api-access-6x96h". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:35:59.631844 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.631804 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7339f829-8832-48c6-ab59-28c69db0f320-error-404-isvc-2138e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:35:59.631844 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.631836 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x96h\" (UniqueName: \"kubernetes.io/projected/7339f829-8832-48c6-ab59-28c69db0f320-kube-api-access-6x96h\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:35:59.631844 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.631846 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7339f829-8832-48c6-ab59-28c69db0f320-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:35:59.704371 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.704337 2567 generic.go:358] "Generic (PLEG): container finished" podID="7339f829-8832-48c6-ab59-28c69db0f320" containerID="d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90" exitCode=0 May 26 16:35:59.704767 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.704408 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" May 26 16:35:59.704767 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.704427 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" event={"ID":"7339f829-8832-48c6-ab59-28c69db0f320","Type":"ContainerDied","Data":"d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90"} May 26 16:35:59.704767 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.704472 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts" event={"ID":"7339f829-8832-48c6-ab59-28c69db0f320","Type":"ContainerDied","Data":"41d8a513deef5e8e00193620e1dd9fc71c0a2f2ed62e8ed35d642ffdde96b03e"} May 26 16:35:59.704767 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.704488 2567 scope.go:117] "RemoveContainer" containerID="0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871" May 26 16:35:59.705064 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.705038 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" May 26 16:35:59.711788 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.711767 2567 scope.go:117] "RemoveContainer" containerID="d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90" May 26 16:35:59.718430 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.718416 2567 scope.go:117] "RemoveContainer" containerID="0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871" May 26 16:35:59.718661 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:35:59.718644 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871\": container with ID starting with 0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871 not found: ID does not exist" containerID="0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871" May 26 16:35:59.718705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.718676 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871"} err="failed to get container status \"0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871\": rpc error: code = NotFound desc = could not find container \"0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871\": container with ID starting with 0c625359f7da7c3779f00eb81631c345a9cf443e963c46b990e84c2e9180c871 not found: ID does not exist" May 26 16:35:59.718705 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.718698 2567 scope.go:117] "RemoveContainer" containerID="d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90" May 26 16:35:59.718900 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:35:59.718886 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90\": container with ID starting with d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90 not found: ID does not exist" containerID="d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90" May 26 16:35:59.718955 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.718904 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90"} err="failed to get container status \"d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90\": rpc error: code = NotFound desc = could not find container \"d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90\": container with ID starting with d8db9349996071ed256c8c4e0a038d74fc9e7d8198ede0a9922c9ff05a055c90 not found: ID does not exist" May 26 16:35:59.727307 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.727285 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts"] May 26 16:35:59.731847 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:35:59.731829 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2138e-predictor-f5c8b557f-z8xts"] May 26 16:36:00.049764 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:00.049732 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7339f829-8832-48c6-ab59-28c69db0f320" path="/var/lib/kubelet/pods/7339f829-8832-48c6-ab59-28c69db0f320/volumes" May 26 16:36:04.709805 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:04.709775 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:36:04.710248 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:04.710224 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" May 26 16:36:14.710307 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:14.710269 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" May 26 16:36:24.710562 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:24.710521 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" May 26 16:36:34.710182 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:34.710142 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" May 26 16:36:35.981896 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:35.981859 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x"] May 26 16:36:35.982390 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:35.982148 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" containerID="cri-o://1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81" gracePeriod=30 May 26 16:36:35.982390 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:35.982232 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kube-rbac-proxy" containerID="cri-o://2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe" gracePeriod=30 May 26 16:36:36.201688 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.201658 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5"] May 26 16:36:36.201966 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.201953 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" May 26 16:36:36.202020 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.201972 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" May 26 16:36:36.202020 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.201996 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kube-rbac-proxy" May 26 16:36:36.202020 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.202003 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kube-rbac-proxy" May 26 16:36:36.202122 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.202059 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kserve-container" May 26 16:36:36.202122 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.202067 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="7339f829-8832-48c6-ab59-28c69db0f320" containerName="kube-rbac-proxy" May 26 16:36:36.204851 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.204832 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.207159 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.207138 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bffff-predictor-serving-cert\"" May 26 16:36:36.207244 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.207159 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bffff-kube-rbac-proxy-sar-config\"" May 26 16:36:36.215912 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.215890 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5"] May 26 16:36:36.299774 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.299691 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb34959-d52b-4733-a39c-f7df77c836f0-proxy-tls\") pod \"error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.299774 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.299761 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb34959-d52b-4733-a39c-f7df77c836f0-error-404-isvc-bffff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.299976 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.299798 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6f9\" (UniqueName: \"kubernetes.io/projected/ecb34959-d52b-4733-a39c-f7df77c836f0-kube-api-access-7l6f9\") pod \"error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.401211 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.401173 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb34959-d52b-4733-a39c-f7df77c836f0-error-404-isvc-bffff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.401384 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.401230 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6f9\" (UniqueName: \"kubernetes.io/projected/ecb34959-d52b-4733-a39c-f7df77c836f0-kube-api-access-7l6f9\") pod \"error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.401384 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.401265 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb34959-d52b-4733-a39c-f7df77c836f0-proxy-tls\") pod \"error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.401902 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.401883 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb34959-d52b-4733-a39c-f7df77c836f0-error-404-isvc-bffff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.403740 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.403723 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb34959-d52b-4733-a39c-f7df77c836f0-proxy-tls\") pod \"error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.410029 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.410003 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6f9\" (UniqueName: \"kubernetes.io/projected/ecb34959-d52b-4733-a39c-f7df77c836f0-kube-api-access-7l6f9\") pod \"error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.515405 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.515367 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.641822 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.641797 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5"] May 26 16:36:36.644277 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:36:36.644253 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb34959_d52b_4733_a39c_f7df77c836f0.slice/crio-697e883c1795adb75164746d44f95131f25852554591d1e69ff09303deee204e WatchSource:0}: Error finding container 697e883c1795adb75164746d44f95131f25852554591d1e69ff09303deee204e: Status 404 returned error can't find the container with id 697e883c1795adb75164746d44f95131f25852554591d1e69ff09303deee204e May 26 16:36:36.806812 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.806780 2567 generic.go:358] "Generic (PLEG): container finished" podID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerID="2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe" exitCode=2 May 26 16:36:36.807001 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.806831 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" event={"ID":"9f0269a7-24fc-4ac2-8813-52425891e08e","Type":"ContainerDied","Data":"2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe"} May 26 16:36:36.808227 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.808206 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" event={"ID":"ecb34959-d52b-4733-a39c-f7df77c836f0","Type":"ContainerStarted","Data":"f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a"} May 26 16:36:36.808325 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.808235 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" event={"ID":"ecb34959-d52b-4733-a39c-f7df77c836f0","Type":"ContainerStarted","Data":"f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025"} May 26 16:36:36.808325 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.808247 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" event={"ID":"ecb34959-d52b-4733-a39c-f7df77c836f0","Type":"ContainerStarted","Data":"697e883c1795adb75164746d44f95131f25852554591d1e69ff09303deee204e"} May 26 16:36:36.808415 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.808403 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:36.827573 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:36.827499 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" podStartSLOduration=0.827484538 podStartE2EDuration="827.484538ms" podCreationTimestamp="2026-05-26 16:36:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:36:36.826747734 +0000 UTC m=+2149.389480679" watchObservedRunningTime="2026-05-26 16:36:36.827484538 +0000 UTC m=+2149.390217481" May 26 16:36:37.811137 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:37.811060 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:37.812262 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:37.812237 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" May 26 16:36:38.813615 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:38.813573 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" May 26 16:36:39.333542 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.333519 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:36:39.422764 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.422731 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0269a7-24fc-4ac2-8813-52425891e08e-error-404-isvc-86373-kube-rbac-proxy-sar-config\") pod \"9f0269a7-24fc-4ac2-8813-52425891e08e\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " May 26 16:36:39.422991 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.422805 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5f7d\" (UniqueName: \"kubernetes.io/projected/9f0269a7-24fc-4ac2-8813-52425891e08e-kube-api-access-t5f7d\") pod \"9f0269a7-24fc-4ac2-8813-52425891e08e\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " May 26 16:36:39.422991 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.422898 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0269a7-24fc-4ac2-8813-52425891e08e-proxy-tls\") pod \"9f0269a7-24fc-4ac2-8813-52425891e08e\" (UID: \"9f0269a7-24fc-4ac2-8813-52425891e08e\") " May 26 16:36:39.423143 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.423121 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0269a7-24fc-4ac2-8813-52425891e08e-error-404-isvc-86373-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-86373-kube-rbac-proxy-sar-config") pod "9f0269a7-24fc-4ac2-8813-52425891e08e" (UID: "9f0269a7-24fc-4ac2-8813-52425891e08e"). InnerVolumeSpecName "error-404-isvc-86373-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:36:39.424908 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.424847 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0269a7-24fc-4ac2-8813-52425891e08e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f0269a7-24fc-4ac2-8813-52425891e08e" (UID: "9f0269a7-24fc-4ac2-8813-52425891e08e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:36:39.424908 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.424849 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0269a7-24fc-4ac2-8813-52425891e08e-kube-api-access-t5f7d" (OuterVolumeSpecName: "kube-api-access-t5f7d") pod "9f0269a7-24fc-4ac2-8813-52425891e08e" (UID: "9f0269a7-24fc-4ac2-8813-52425891e08e"). InnerVolumeSpecName "kube-api-access-t5f7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:36:39.524401 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.524365 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0269a7-24fc-4ac2-8813-52425891e08e-error-404-isvc-86373-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:36:39.524401 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.524396 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5f7d\" (UniqueName: \"kubernetes.io/projected/9f0269a7-24fc-4ac2-8813-52425891e08e-kube-api-access-t5f7d\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:36:39.524401 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.524406 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0269a7-24fc-4ac2-8813-52425891e08e-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:36:39.817740 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.817657 2567 generic.go:358] "Generic (PLEG): container finished" podID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerID="1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81" exitCode=0 May 26 16:36:39.817740 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.817724 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" event={"ID":"9f0269a7-24fc-4ac2-8813-52425891e08e","Type":"ContainerDied","Data":"1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81"} May 26 16:36:39.818245 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.817755 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" event={"ID":"9f0269a7-24fc-4ac2-8813-52425891e08e","Type":"ContainerDied","Data":"58fb5b618f341f5b25c54cf490ca8e2b4a42fff3b4e211c48d206c5b6d24c906"} May 26 16:36:39.818245 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.817776 2567 scope.go:117] "RemoveContainer" containerID="2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe" May 26 16:36:39.818245 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.817726 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" May 26 16:36:39.826179 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.826141 2567 scope.go:117] "RemoveContainer" containerID="1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81" May 26 16:36:39.832810 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.832794 2567 scope.go:117] "RemoveContainer" containerID="2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe" May 26 16:36:39.833073 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:36:39.833051 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe\": container with ID starting with 2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe not found: ID does not exist" containerID="2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe" May 26 16:36:39.833130 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.833082 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe"} err="failed to get container status \"2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe\": rpc error: code = NotFound desc = could not find container \"2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe\": container with ID starting with 2c33918a72a1b840b3014742d4fe4c1f138a158f8efae7b85446775f3d2385fe not found: ID does not exist" May 26 16:36:39.833130 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.833099 2567 scope.go:117] "RemoveContainer" containerID="1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81" May 26 16:36:39.833317 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:36:39.833297 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81\": container with ID starting with 1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81 not found: ID does not exist" containerID="1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81" May 26 16:36:39.833394 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.833322 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81"} err="failed to get container status \"1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81\": rpc error: code = NotFound desc = could not find container \"1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81\": container with ID starting with 1b60953b04a616461e07f338a63d337347ac54ebeef9a237d5543600a9e4df81 not found: ID does not exist" May 26 16:36:39.840254 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.840229 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x"] May 26 16:36:39.844670 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:39.844648 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x"] May 26 16:36:40.049959 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:40.049909 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" path="/var/lib/kubelet/pods/9f0269a7-24fc-4ac2-8813-52425891e08e/volumes" May 26 16:36:40.331391 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:40.331352 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86373-predictor-776c7fd767-prf4x" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" May 26 16:36:43.817646 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:43.817620 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:36:43.818266 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:43.818245 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" May 26 16:36:44.711064 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:44.711033 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:36:53.818528 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:36:53.818488 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" May 26 16:37:03.818312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:03.818266 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" May 26 16:37:06.633464 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.633432 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp"] May 26 16:37:06.633836 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.633740 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" containerID="cri-o://bd1b191ecdc88469428e5652dfa9e847b7e18063d2f4d1861949911561742cb8" gracePeriod=30 May 26 16:37:06.633836 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.633802 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kube-rbac-proxy" containerID="cri-o://04365e47ee340172d2ea78accb76f1324cf396f07ae6a7aba4a39222aaa10ef0" gracePeriod=30 May 26 16:37:06.702383 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.702343 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9"] May 26 16:37:06.702683 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.702670 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kube-rbac-proxy" May 26 16:37:06.702727 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.702686 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kube-rbac-proxy" May 26 16:37:06.702727 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.702705 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" May 26 16:37:06.702727 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.702712 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" May 26 16:37:06.702825 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.702771 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kube-rbac-proxy" May 26 16:37:06.702825 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.702786 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="9f0269a7-24fc-4ac2-8813-52425891e08e" containerName="kserve-container" May 26 16:37:06.705926 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.705905 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:06.708204 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.708182 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5798a-kube-rbac-proxy-sar-config\"" May 26 16:37:06.708442 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.708426 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5798a-predictor-serving-cert\"" May 26 16:37:06.713725 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.713701 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9"] May 26 16:37:06.833550 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.833517 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmn2\" (UniqueName: \"kubernetes.io/projected/b7950198-cec3-4406-8d62-6befd91ffa5d-kube-api-access-9pmn2\") pod \"error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:06.833713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.833564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7950198-cec3-4406-8d62-6befd91ffa5d-proxy-tls\") pod \"error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:06.833713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.833592 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7950198-cec3-4406-8d62-6befd91ffa5d-error-404-isvc-5798a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:06.889853 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.889759 2567 generic.go:358] "Generic (PLEG): container finished" podID="75743ec9-c617-4958-a564-df9910f039a5" containerID="04365e47ee340172d2ea78accb76f1324cf396f07ae6a7aba4a39222aaa10ef0" exitCode=2 May 26 16:37:06.889853 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.889834 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" event={"ID":"75743ec9-c617-4958-a564-df9910f039a5","Type":"ContainerDied","Data":"04365e47ee340172d2ea78accb76f1324cf396f07ae6a7aba4a39222aaa10ef0"} May 26 16:37:06.934314 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.934266 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmn2\" (UniqueName: \"kubernetes.io/projected/b7950198-cec3-4406-8d62-6befd91ffa5d-kube-api-access-9pmn2\") pod \"error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:06.934428 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.934336 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7950198-cec3-4406-8d62-6befd91ffa5d-proxy-tls\") pod \"error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:06.934428 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.934359 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7950198-cec3-4406-8d62-6befd91ffa5d-error-404-isvc-5798a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:06.935034 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.935011 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7950198-cec3-4406-8d62-6befd91ffa5d-error-404-isvc-5798a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:06.936727 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.936708 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7950198-cec3-4406-8d62-6befd91ffa5d-proxy-tls\") pod \"error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:06.943211 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:06.943178 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmn2\" (UniqueName: \"kubernetes.io/projected/b7950198-cec3-4406-8d62-6befd91ffa5d-kube-api-access-9pmn2\") pod \"error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:07.017695 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:07.017653 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:07.144468 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:07.144385 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9"] May 26 16:37:07.147973 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:37:07.147947 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7950198_cec3_4406_8d62_6befd91ffa5d.slice/crio-58aab420969b71fffa64d437c953293d97d23eb1770b0571e4ba743c9b9b636c WatchSource:0}: Error finding container 58aab420969b71fffa64d437c953293d97d23eb1770b0571e4ba743c9b9b636c: Status 404 returned error can't find the container with id 58aab420969b71fffa64d437c953293d97d23eb1770b0571e4ba743c9b9b636c May 26 16:37:07.894003 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:07.893965 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" event={"ID":"b7950198-cec3-4406-8d62-6befd91ffa5d","Type":"ContainerStarted","Data":"aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be"} May 26 16:37:07.894003 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:07.894005 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" event={"ID":"b7950198-cec3-4406-8d62-6befd91ffa5d","Type":"ContainerStarted","Data":"9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a"} May 26 16:37:07.894433 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:07.894016 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" event={"ID":"b7950198-cec3-4406-8d62-6befd91ffa5d","Type":"ContainerStarted","Data":"58aab420969b71fffa64d437c953293d97d23eb1770b0571e4ba743c9b9b636c"} May 26 16:37:07.894433 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:07.894108 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:07.914662 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:07.914616 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" podStartSLOduration=1.91460243 podStartE2EDuration="1.91460243s" podCreationTimestamp="2026-05-26 16:37:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:37:07.913883425 +0000 UTC m=+2180.476616369" watchObservedRunningTime="2026-05-26 16:37:07.91460243 +0000 UTC m=+2180.477335373" May 26 16:37:08.896729 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:08.896694 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:08.897752 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:08.897727 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" May 26 16:37:09.706068 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:09.706020 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused" May 26 16:37:09.901215 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:09.901184 2567 generic.go:358] "Generic (PLEG): container finished" podID="75743ec9-c617-4958-a564-df9910f039a5" containerID="bd1b191ecdc88469428e5652dfa9e847b7e18063d2f4d1861949911561742cb8" exitCode=0 May 26 16:37:09.901575 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:09.901255 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" event={"ID":"75743ec9-c617-4958-a564-df9910f039a5","Type":"ContainerDied","Data":"bd1b191ecdc88469428e5652dfa9e847b7e18063d2f4d1861949911561742cb8"} May 26 16:37:09.901713 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:09.901682 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" May 26 16:37:09.979157 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:09.979132 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:37:10.059671 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.059626 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75743ec9-c617-4958-a564-df9910f039a5-proxy-tls\") pod \"75743ec9-c617-4958-a564-df9910f039a5\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " May 26 16:37:10.059837 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.059692 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq77m\" (UniqueName: \"kubernetes.io/projected/75743ec9-c617-4958-a564-df9910f039a5-kube-api-access-vq77m\") pod \"75743ec9-c617-4958-a564-df9910f039a5\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " May 26 16:37:10.059837 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.059790 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75743ec9-c617-4958-a564-df9910f039a5-error-404-isvc-82fbc-kube-rbac-proxy-sar-config\") pod \"75743ec9-c617-4958-a564-df9910f039a5\" (UID: \"75743ec9-c617-4958-a564-df9910f039a5\") " May 26 16:37:10.060204 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.060172 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75743ec9-c617-4958-a564-df9910f039a5-error-404-isvc-82fbc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-82fbc-kube-rbac-proxy-sar-config") pod "75743ec9-c617-4958-a564-df9910f039a5" (UID: "75743ec9-c617-4958-a564-df9910f039a5"). InnerVolumeSpecName "error-404-isvc-82fbc-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:37:10.062217 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.062197 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75743ec9-c617-4958-a564-df9910f039a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75743ec9-c617-4958-a564-df9910f039a5" (UID: "75743ec9-c617-4958-a564-df9910f039a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:37:10.062313 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.062222 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75743ec9-c617-4958-a564-df9910f039a5-kube-api-access-vq77m" (OuterVolumeSpecName: "kube-api-access-vq77m") pod "75743ec9-c617-4958-a564-df9910f039a5" (UID: "75743ec9-c617-4958-a564-df9910f039a5"). InnerVolumeSpecName "kube-api-access-vq77m". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:37:10.160558 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.160517 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75743ec9-c617-4958-a564-df9910f039a5-error-404-isvc-82fbc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:37:10.160558 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.160551 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75743ec9-c617-4958-a564-df9910f039a5-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:37:10.160558 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.160565 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq77m\" (UniqueName: \"kubernetes.io/projected/75743ec9-c617-4958-a564-df9910f039a5-kube-api-access-vq77m\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:37:10.905425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.905388 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" event={"ID":"75743ec9-c617-4958-a564-df9910f039a5","Type":"ContainerDied","Data":"78cf6aea96cbb7e1a650152b8f2631fff3ec7ad8031bb0a3ca1c437410caeb62"} May 26 16:37:10.905425 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.905433 2567 scope.go:117] "RemoveContainer" containerID="04365e47ee340172d2ea78accb76f1324cf396f07ae6a7aba4a39222aaa10ef0" May 26 16:37:10.906017 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.905402 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp" May 26 16:37:10.913962 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.913599 2567 scope.go:117] "RemoveContainer" containerID="bd1b191ecdc88469428e5652dfa9e847b7e18063d2f4d1861949911561742cb8" May 26 16:37:10.928898 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.928866 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp"] May 26 16:37:10.935733 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:10.935706 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-82fbc-predictor-855f648bc-wjldp"] May 26 16:37:12.049557 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:12.049510 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75743ec9-c617-4958-a564-df9910f039a5" path="/var/lib/kubelet/pods/75743ec9-c617-4958-a564-df9910f039a5/volumes" May 26 16:37:13.818595 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:13.818556 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" May 26 16:37:14.906029 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:14.905996 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:37:14.906586 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:14.906557 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" May 26 16:37:23.819102 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:23.819071 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:37:24.906799 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:24.906750 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" May 26 16:37:34.906500 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:34.906461 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" May 26 16:37:44.906645 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:44.906600 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" May 26 16:37:54.907085 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:37:54.907052 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:40:48.115430 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:40:48.115400 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:40:48.120001 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:40:48.119983 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:45:48.132691 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:45:48.132590 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:45:48.138233 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:45:48.138214 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:46:21.444240 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:21.444204 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9"] May 26 16:46:21.444781 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:21.444479 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" containerID="cri-o://9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a" gracePeriod=30 May 26 16:46:21.444781 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:21.444533 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kube-rbac-proxy" containerID="cri-o://aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be" gracePeriod=30 May 26 16:46:22.401183 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:22.401151 2567 generic.go:358] "Generic (PLEG): container finished" podID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerID="aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be" exitCode=2 May 26 16:46:22.401350 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:22.401194 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" event={"ID":"b7950198-cec3-4406-8d62-6befd91ffa5d","Type":"ContainerDied","Data":"aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be"} May 26 16:46:24.394151 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.394127 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:46:24.409971 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.409943 2567 generic.go:358] "Generic (PLEG): container finished" podID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerID="9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a" exitCode=0 May 26 16:46:24.410121 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.410024 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" May 26 16:46:24.410121 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.410027 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" event={"ID":"b7950198-cec3-4406-8d62-6befd91ffa5d","Type":"ContainerDied","Data":"9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a"} May 26 16:46:24.410221 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.410129 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9" event={"ID":"b7950198-cec3-4406-8d62-6befd91ffa5d","Type":"ContainerDied","Data":"58aab420969b71fffa64d437c953293d97d23eb1770b0571e4ba743c9b9b636c"} May 26 16:46:24.410221 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.410145 2567 scope.go:117] "RemoveContainer" containerID="aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be" May 26 16:46:24.417620 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.417583 2567 scope.go:117] "RemoveContainer" containerID="9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a" May 26 16:46:24.424584 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.424562 2567 scope.go:117] "RemoveContainer" containerID="aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be" May 26 16:46:24.424818 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:46:24.424800 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be\": container with ID starting with aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be not found: ID does not exist" containerID="aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be" May 26 16:46:24.424874 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.424825 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be"} err="failed to get container status \"aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be\": rpc error: code = NotFound desc = could not find container \"aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be\": container with ID starting with aa4284603b17a90292cd0fbcc2f48b52feffad1f6193f9101c4dc59c13b983be not found: ID does not exist" May 26 16:46:24.424874 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.424846 2567 scope.go:117] "RemoveContainer" containerID="9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a" May 26 16:46:24.425155 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:46:24.425129 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a\": container with ID starting with 9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a not found: ID does not exist" containerID="9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a" May 26 16:46:24.425269 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.425160 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a"} err="failed to get container status \"9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a\": rpc error: code = NotFound desc = could not find container \"9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a\": container with ID starting with 9a03b918a149f51011f1f0cf1c902ac5499780b7bf9fffe3bb40d439cd3df36a not found: ID does not exist" May 26 16:46:24.540739 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.540650 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pmn2\" (UniqueName: \"kubernetes.io/projected/b7950198-cec3-4406-8d62-6befd91ffa5d-kube-api-access-9pmn2\") pod \"b7950198-cec3-4406-8d62-6befd91ffa5d\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " May 26 16:46:24.540739 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.540706 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7950198-cec3-4406-8d62-6befd91ffa5d-proxy-tls\") pod \"b7950198-cec3-4406-8d62-6befd91ffa5d\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " May 26 16:46:24.540739 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.540731 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7950198-cec3-4406-8d62-6befd91ffa5d-error-404-isvc-5798a-kube-rbac-proxy-sar-config\") pod \"b7950198-cec3-4406-8d62-6befd91ffa5d\" (UID: \"b7950198-cec3-4406-8d62-6befd91ffa5d\") " May 26 16:46:24.541138 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.541114 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7950198-cec3-4406-8d62-6befd91ffa5d-error-404-isvc-5798a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-5798a-kube-rbac-proxy-sar-config") pod "b7950198-cec3-4406-8d62-6befd91ffa5d" (UID: "b7950198-cec3-4406-8d62-6befd91ffa5d"). InnerVolumeSpecName "error-404-isvc-5798a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:46:24.542762 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.542735 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7950198-cec3-4406-8d62-6befd91ffa5d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7950198-cec3-4406-8d62-6befd91ffa5d" (UID: "b7950198-cec3-4406-8d62-6befd91ffa5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:46:24.542873 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.542800 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7950198-cec3-4406-8d62-6befd91ffa5d-kube-api-access-9pmn2" (OuterVolumeSpecName: "kube-api-access-9pmn2") pod "b7950198-cec3-4406-8d62-6befd91ffa5d" (UID: "b7950198-cec3-4406-8d62-6befd91ffa5d"). InnerVolumeSpecName "kube-api-access-9pmn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:46:24.641666 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.641632 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7950198-cec3-4406-8d62-6befd91ffa5d-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:46:24.641666 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.641661 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7950198-cec3-4406-8d62-6befd91ffa5d-error-404-isvc-5798a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:46:24.641666 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.641672 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmn2\" (UniqueName: \"kubernetes.io/projected/b7950198-cec3-4406-8d62-6befd91ffa5d-kube-api-access-9pmn2\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:46:24.734907 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.734876 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9"] May 26 16:46:24.737366 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:24.737343 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5798a-predictor-7c9d6c8d5c-xjxw9"] May 26 16:46:26.049699 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:46:26.049665 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" path="/var/lib/kubelet/pods/b7950198-cec3-4406-8d62-6befd91ffa5d/volumes" May 26 16:50:48.149248 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:50:48.149131 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:50:48.156254 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:50:48.156236 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:53:55.450861 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:55.450826 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5"] May 26 16:53:55.451481 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:55.451139 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" containerID="cri-o://f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025" gracePeriod=30 May 26 16:53:55.451481 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:55.451180 2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kube-rbac-proxy" containerID="cri-o://f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a" gracePeriod=30 May 26 16:53:55.576834 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:55.576805 2567 generic.go:358] "Generic (PLEG): container finished" podID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerID="f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a" exitCode=2 May 26 16:53:55.576989 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:55.576875 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" event={"ID":"ecb34959-d52b-4733-a39c-f7df77c836f0","Type":"ContainerDied","Data":"f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a"} May 26 16:53:58.290878 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.290855 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:53:58.470131 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.470094 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb34959-d52b-4733-a39c-f7df77c836f0-error-404-isvc-bffff-kube-rbac-proxy-sar-config\") pod \"ecb34959-d52b-4733-a39c-f7df77c836f0\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " May 26 16:53:58.470131 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.470133 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l6f9\" (UniqueName: \"kubernetes.io/projected/ecb34959-d52b-4733-a39c-f7df77c836f0-kube-api-access-7l6f9\") pod \"ecb34959-d52b-4733-a39c-f7df77c836f0\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " May 26 16:53:58.470332 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.470181 2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb34959-d52b-4733-a39c-f7df77c836f0-proxy-tls\") pod \"ecb34959-d52b-4733-a39c-f7df77c836f0\" (UID: \"ecb34959-d52b-4733-a39c-f7df77c836f0\") " May 26 16:53:58.470476 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.470451 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb34959-d52b-4733-a39c-f7df77c836f0-error-404-isvc-bffff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-bffff-kube-rbac-proxy-sar-config") pod "ecb34959-d52b-4733-a39c-f7df77c836f0" (UID: "ecb34959-d52b-4733-a39c-f7df77c836f0"). InnerVolumeSpecName "error-404-isvc-bffff-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:53:58.472278 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.472248 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb34959-d52b-4733-a39c-f7df77c836f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecb34959-d52b-4733-a39c-f7df77c836f0" (UID: "ecb34959-d52b-4733-a39c-f7df77c836f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:53:58.472361 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.472289 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb34959-d52b-4733-a39c-f7df77c836f0-kube-api-access-7l6f9" (OuterVolumeSpecName: "kube-api-access-7l6f9") pod "ecb34959-d52b-4733-a39c-f7df77c836f0" (UID: "ecb34959-d52b-4733-a39c-f7df77c836f0"). InnerVolumeSpecName "kube-api-access-7l6f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:53:58.570816 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.570776 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb34959-d52b-4733-a39c-f7df77c836f0-proxy-tls\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:53:58.570816 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.570814 2567 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb34959-d52b-4733-a39c-f7df77c836f0-error-404-isvc-bffff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:53:58.570816 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.570823 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6f9\" (UniqueName: \"kubernetes.io/projected/ecb34959-d52b-4733-a39c-f7df77c836f0-kube-api-access-7l6f9\") on node \"ip-10-0-131-126.ec2.internal\" DevicePath \"\"" May 26 16:53:58.585392 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.585361 2567 generic.go:358] "Generic (PLEG): container finished" podID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerID="f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025" exitCode=0 May 26 16:53:58.585517 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.585436 2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" May 26 16:53:58.585517 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.585436 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" event={"ID":"ecb34959-d52b-4733-a39c-f7df77c836f0","Type":"ContainerDied","Data":"f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025"} May 26 16:53:58.585517 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.585481 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5" event={"ID":"ecb34959-d52b-4733-a39c-f7df77c836f0","Type":"ContainerDied","Data":"697e883c1795adb75164746d44f95131f25852554591d1e69ff09303deee204e"} May 26 16:53:58.585517 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.585501 2567 scope.go:117] "RemoveContainer" containerID="f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a" May 26 16:53:58.593995 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.593971 2567 scope.go:117] "RemoveContainer" containerID="f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025" May 26 16:53:58.603612 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.603594 2567 scope.go:117] "RemoveContainer" containerID="f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a" May 26 16:53:58.603888 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:53:58.603859 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a\": container with ID starting with f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a not found: ID does not exist" containerID="f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a" May 26 16:53:58.603984 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.603898 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a"} err="failed to get container status \"f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a\": rpc error: code = NotFound desc = could not find container \"f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a\": container with ID starting with f949326dfc179648f7ea56a4e66058b9de845ccf59ec852a782e578f7f0d440a not found: ID does not exist" May 26 16:53:58.603984 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.603925 2567 scope.go:117] "RemoveContainer" containerID="f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025" May 26 16:53:58.604203 ip-10-0-131-126 kubenswrapper[2567]: E0526 16:53:58.604167 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025\": container with ID starting with f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025 not found: ID does not exist" containerID="f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025" May 26 16:53:58.604257 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.604213 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025"} err="failed to get container status \"f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025\": rpc error: code = NotFound desc = could not find container \"f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025\": container with ID starting with f6b929cdd6649c8c40719f787e8a4212bfdf130480d51e6115aea3e2648f1025 not found: ID does not exist" May 26 16:53:58.607204 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.607183 2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5"] May 26 16:53:58.612001 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:53:58.611979 2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bffff-predictor-6c4898b5bf-4fsw5"] May 26 16:54:00.049411 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:00.049381 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" path="/var/lib/kubelet/pods/ecb34959-d52b-4733-a39c-f7df77c836f0/volumes" May 26 16:54:30.131718 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:30.131684 2567 ???:1] "http: TLS handshake error from 10.0.129.89:39234: EOF" May 26 16:54:30.139436 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:30.139413 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6nfnx_3874d0e8-75c4-4fed-be1a-f305a44abfa8/global-pull-secret-syncer/0.log" May 26 16:54:30.322400 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:30.322370 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qg7wc_d21ccc94-60a4-4663-bbf1-a243a95bfa44/konnectivity-agent/0.log" May 26 16:54:30.398905 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:30.398795 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-126.ec2.internal_4f187ab729b0a5371eb7618f4a50c541/haproxy/0.log" May 26 16:54:34.289182 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:34.289151 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm94q_0d7de434-2277-41a1-bb6c-85e0255218b7/node-exporter/0.log" May 26 16:54:34.312300 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:34.312278 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm94q_0d7de434-2277-41a1-bb6c-85e0255218b7/kube-rbac-proxy/0.log" May 26 16:54:34.335978 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:34.335955 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm94q_0d7de434-2277-41a1-bb6c-85e0255218b7/init-textfile/0.log" May 26 16:54:36.123365 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:36.123331 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-dwthz_b8aee9c3-623c-4438-934c-ea0e437eda7a/networking-console-plugin/0.log" May 26 16:54:37.315904 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.315823 2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc"] May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316191 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316209 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316225 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kube-rbac-proxy" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316230 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kube-rbac-proxy" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316241 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316247 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316253 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316259 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316267 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kube-rbac-proxy" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316273 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kube-rbac-proxy" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316280 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kube-rbac-proxy" May 26 16:54:37.316312 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316285 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kube-rbac-proxy" May 26 16:54:37.316803 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316325 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kserve-container" May 26 16:54:37.316803 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316333 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kube-rbac-proxy" May 26 16:54:37.316803 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316340 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="ecb34959-d52b-4733-a39c-f7df77c836f0" containerName="kube-rbac-proxy" May 26 16:54:37.316803 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316346 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="b7950198-cec3-4406-8d62-6befd91ffa5d" containerName="kserve-container" May 26 16:54:37.316803 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316353 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kserve-container" May 26 16:54:37.316803 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.316360 2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="75743ec9-c617-4958-a564-df9910f039a5" containerName="kube-rbac-proxy" May 26 16:54:37.319403 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.319387 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.321711 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.321688 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sh5kf\"/\"openshift-service-ca.crt\"" May 26 16:54:37.321802 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.321711 2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sh5kf\"/\"kube-root-ca.crt\"" May 26 16:54:37.322482 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.322469 2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sh5kf\"/\"default-dockercfg-hlrkd\"" May 26 16:54:37.327032 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.327010 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc"] May 26 16:54:37.344869 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.344841 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-sys\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.345010 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.344873 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svhx4\" (UniqueName: \"kubernetes.io/projected/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-kube-api-access-svhx4\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.345010 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.344916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-lib-modules\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.345010 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.344959 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-podres\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.345010 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.344983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-proc\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.445545 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.445510 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-lib-modules\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.445545 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.445548 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-podres\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.445777 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.445571 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-proc\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.445777 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.445631 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-proc\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.445777 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.445657 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-sys\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.445777 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.445679 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-lib-modules\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.445777 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.445686 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-podres\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.445777 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.445704 2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-svhx4\" (UniqueName: \"kubernetes.io/projected/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-kube-api-access-svhx4\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.445777 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.445707 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-sys\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.454207 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.454181 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhx4\" (UniqueName: \"kubernetes.io/projected/a24df79a-8cf0-4bf0-a935-9a06d7ae602f-kube-api-access-svhx4\") pod \"perf-node-gather-daemonset-qjczc\" (UID: \"a24df79a-8cf0-4bf0-a935-9a06d7ae602f\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.629673 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.629634 2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:37.753203 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.753176 2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc"] May 26 16:54:37.755559 ip-10-0-131-126 kubenswrapper[2567]: W0526 16:54:37.755526 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda24df79a_8cf0_4bf0_a935_9a06d7ae602f.slice/crio-3d3f39f6b293760e81f8e9b64fba29b580f17e2ca610b8ba8f17947ef35986ee WatchSource:0}: Error finding container 3d3f39f6b293760e81f8e9b64fba29b580f17e2ca610b8ba8f17947ef35986ee: Status 404 returned error can't find the container with id 3d3f39f6b293760e81f8e9b64fba29b580f17e2ca610b8ba8f17947ef35986ee May 26 16:54:37.757081 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:37.757063 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:54:38.067220 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:38.067134 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2kl9l_70fb5ba8-e654-4f66-8a9a-597dec2fde25/dns/0.log" May 26 16:54:38.090166 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:38.090140 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2kl9l_70fb5ba8-e654-4f66-8a9a-597dec2fde25/kube-rbac-proxy/0.log" May 26 16:54:38.215584 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:38.215553 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-czhdz_3a30554b-dd8f-4aa1-a4ec-e4bc64b466fe/dns-node-resolver/0.log" May 26 16:54:38.693037 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:38.692999 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" event={"ID":"a24df79a-8cf0-4bf0-a935-9a06d7ae602f","Type":"ContainerStarted","Data":"782741fbe4f6ff845e1b3ca36ecbe092971f379777d39ba853a5b92881370cea"} May 26 16:54:38.693037 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:38.693038 2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" event={"ID":"a24df79a-8cf0-4bf0-a935-9a06d7ae602f","Type":"ContainerStarted","Data":"3d3f39f6b293760e81f8e9b64fba29b580f17e2ca610b8ba8f17947ef35986ee"} May 26 16:54:38.693465 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:38.693130 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:38.710738 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:38.710695 2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" podStartSLOduration=1.710681236 podStartE2EDuration="1.710681236s" podCreationTimestamp="2026-05-26 16:54:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:54:38.708814214 +0000 UTC m=+3231.271547156" watchObservedRunningTime="2026-05-26 16:54:38.710681236 +0000 UTC m=+3231.273414179" May 26 16:54:38.791561 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:38.791531 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q7xm7_77d4a48a-9687-4338-b49d-1219168b52e2/node-ca/0.log" May 26 16:54:39.861665 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:39.861628 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rl5nn_86cf44b2-9266-4e01-a88a-b1552e2ae1bd/serve-healthcheck-canary/0.log" May 26 16:54:40.359186 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:40.359157 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mgdrc_c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5/kube-rbac-proxy/0.log" May 26 16:54:40.380811 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:40.380780 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mgdrc_c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5/exporter/0.log" May 26 16:54:40.408883 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:40.408845 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mgdrc_c6bf2db6-ca7f-47b5-bba6-fabedaca7fe5/extractor/0.log" May 26 16:54:42.377341 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:42.377302 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6f7545b5b9-j9b4s_1c3ccead-1026-4038-bdae-b92ff3744e7c/manager/0.log" May 26 16:54:42.424858 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:42.424825 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-gxx84_82753df3-8c5c-49a6-89a3-bdea2cfd83f8/server/0.log" May 26 16:54:44.705022 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:44.704993 2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qjczc" May 26 16:54:48.574214 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.574138 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2msh_442af588-81ae-4c16-bafd-6a466acbaf11/kube-multus-additional-cni-plugins/0.log" May 26 16:54:48.596131 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.596102 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2msh_442af588-81ae-4c16-bafd-6a466acbaf11/egress-router-binary-copy/0.log" May 26 16:54:48.619659 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.619630 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2msh_442af588-81ae-4c16-bafd-6a466acbaf11/cni-plugins/0.log" May 26 16:54:48.643529 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.643509 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2msh_442af588-81ae-4c16-bafd-6a466acbaf11/bond-cni-plugin/0.log" May 26 16:54:48.664431 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.664400 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2msh_442af588-81ae-4c16-bafd-6a466acbaf11/routeoverride-cni/0.log" May 26 16:54:48.687336 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.687317 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2msh_442af588-81ae-4c16-bafd-6a466acbaf11/whereabouts-cni-bincopy/0.log" May 26 16:54:48.709257 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.709225 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2msh_442af588-81ae-4c16-bafd-6a466acbaf11/whereabouts-cni/0.log" May 26 16:54:48.775145 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.775118 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l6rc8_dbcac5bd-82bf-42d1-a0a4-61734ba3a6ee/kube-multus/0.log" May 26 16:54:48.929218 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.929191 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x7p82_ddf6bd23-ce8a-4927-b11f-c552d481bc33/network-metrics-daemon/0.log" May 26 16:54:48.948393 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:48.948367 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x7p82_ddf6bd23-ce8a-4927-b11f-c552d481bc33/kube-rbac-proxy/0.log" May 26 16:54:50.341318 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:50.341290 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-controller/0.log" May 26 16:54:50.383447 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:50.383416 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/0.log" May 26 16:54:50.397378 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:50.397353 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovn-acl-logging/1.log" May 26 16:54:50.424949 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:50.424912 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/kube-rbac-proxy-node/0.log" May 26 16:54:50.456862 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:50.456834 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/kube-rbac-proxy-ovn-metrics/0.log" May 26 16:54:50.486775 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:50.486747 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/northd/0.log" May 26 16:54:50.513576 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:50.513550 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/nbdb/0.log" May 26 16:54:50.535455 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:50.535432 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/sbdb/0.log" May 26 16:54:50.627415 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:50.627384 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vflns_cec49982-8c45-4e9a-a2f1-9d7b6bf64030/ovnkube-controller/0.log" May 26 16:54:51.689200 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:51.689170 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-r6sp2_86a247ed-5b25-4bd5-86f5-4ebca2a54f44/network-check-target-container/0.log" May 26 16:54:52.592102 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:52.592076 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-p72d6_3f256df7-c5ad-4a87-8369-dace4f5d89d1/iptables-alerter/0.log" May 26 16:54:53.213172 ip-10-0-131-126 kubenswrapper[2567]: I0526 16:54:53.213144 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-t495h_7b7040cd-6bfe-4c91-aa1a-59981592b6f2/tuned/0.log"