May 26 16:00:51.019140 ip-10-0-140-48 systemd[1]: Starting Kubernetes Kubelet... May 26 16:00:51.593293 ip-10-0-140-48 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 26 16:00:51.593293 ip-10-0-140-48 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. May 26 16:00:51.593293 ip-10-0-140-48 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 26 16:00:51.593293 ip-10-0-140-48 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. May 26 16:00:51.593293 ip-10-0-140-48 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 26 16:00:51.596300 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.596209 2573 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" May 26 16:00:51.602753 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602738 2573 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:51.602753 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602753 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:51.602753 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602757 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602760 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602765 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602768 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602771 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602774 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602783 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602785 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602788 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602791 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602793 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602796 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602798 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602801 2573 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602803 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602806 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602809 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602812 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602814 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602817 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:51.602861 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602819 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602822 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602824 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602840 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602843 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602845 2573 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602848 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602851 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602853 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602856 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602858 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602863 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602867 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602870 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602873 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602876 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602879 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602882 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602884 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602887 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:51.603333 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602889 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602893 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602895 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602897 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602900 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602903 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602905 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602908 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602910 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602913 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602916 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602918 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602921 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602923 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602927 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602930 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602933 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602937 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602940 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:51.603822 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602943 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602945 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602949 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602951 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602953 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602958 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602962 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602965 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602968 2573 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602973 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602976 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602979 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602981 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602984 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602987 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602989 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602992 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602995 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.602998 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:51.604283 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603001 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603003 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603005 2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603008 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603010 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603013 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603406 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603411 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603414 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603417 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603420 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603422 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603425 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603428 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603438 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603442 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603445 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603447 2573 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603451 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603454 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:51.604717 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603457 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603459 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603464 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603468 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603471 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603474 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603478 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603482 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603485 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603488 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603490 2573 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603493 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603495 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603498 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603500 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603503 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603506 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603508 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603511 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:51.605205 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603513 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603517 2573 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603520 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603522 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603525 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603527 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603530 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603532 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603535 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603538 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603540 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603543 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603546 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603548 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603551 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603553 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603557 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603559 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603562 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603564 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:51.605660 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603567 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603569 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603572 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603574 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603577 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603580 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603583 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603585 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603588 2573 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603590 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603593 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603595 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603598 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603603 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603605 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603608 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603610 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603613 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603615 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603618 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:51.606154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603620 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603623 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603625 2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603628 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603630 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603633 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603635 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603638 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603641 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603645 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603647 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603650 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.603652 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604485 2573 flags.go:64] FLAG: --address="0.0.0.0" May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604494 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604501 2573 flags.go:64] FLAG: --anonymous-auth="true" May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604506 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604510 2573 flags.go:64] FLAG: --authentication-token-webhook="false" May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604513 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604517 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604527 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" May 26 16:00:51.606668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604530 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604533 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604537 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604540 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604544 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604548 2573 flags.go:64] FLAG: --cgroup-root="" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604551 2573 flags.go:64] FLAG: --cgroups-per-qos="true" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604554 2573 flags.go:64] FLAG: --client-ca-file="" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604557 2573 flags.go:64] FLAG: --cloud-config="" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604560 2573 flags.go:64] FLAG: --cloud-provider="external" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604563 2573 flags.go:64] FLAG: --cluster-dns="[]" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604568 2573 flags.go:64] FLAG: --cluster-domain="" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604571 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604574 2573 flags.go:64] FLAG: --config-dir="" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604577 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604580 2573 flags.go:64] FLAG: --container-log-max-files="5" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604584 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604588 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604591 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604594 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604597 2573 flags.go:64] FLAG: --contention-profiling="false" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604601 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604604 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604607 2573 flags.go:64] FLAG: --cpu-manager-policy="none" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604610 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" May 26 16:00:51.607184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604614 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604617 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604620 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604623 2573 flags.go:64] FLAG: --enable-load-reader="false" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604626 2573 flags.go:64] FLAG: --enable-server="true" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604629 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604633 2573 flags.go:64] FLAG: --event-burst="100" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604637 2573 flags.go:64] FLAG: --event-qps="50" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604640 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604643 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604646 2573 flags.go:64] FLAG: --eviction-hard="" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604650 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604653 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604656 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604659 2573 flags.go:64] FLAG: --eviction-soft="" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604662 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604665 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604668 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604671 2573 flags.go:64] FLAG: --experimental-mounter-path="" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604674 2573 flags.go:64] FLAG: --fail-cgroupv1="false" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604677 2573 flags.go:64] FLAG: --fail-swap-on="true" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604680 2573 flags.go:64] FLAG: --feature-gates="" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604684 2573 flags.go:64] FLAG: --file-check-frequency="20s" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604687 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604690 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" May 26 16:00:51.607795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604694 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604697 2573 flags.go:64] FLAG: --healthz-port="10248" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604700 2573 flags.go:64] FLAG: --help="false" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604703 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-140-48.ec2.internal" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604707 2573 flags.go:64] FLAG: --housekeeping-interval="10s" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604710 2573 flags.go:64] FLAG: --http-check-frequency="20s" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604713 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604717 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604721 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604724 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604727 2573 flags.go:64] FLAG: --image-service-endpoint="" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604730 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604733 2573 flags.go:64] FLAG: --kube-api-burst="100" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604736 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604739 2573 flags.go:64] FLAG: --kube-api-qps="50" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604743 2573 flags.go:64] FLAG: --kube-reserved="" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604746 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604749 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604752 2573 flags.go:64] FLAG: --kubelet-cgroups="" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604754 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604757 2573 flags.go:64] FLAG: --lock-file="" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604760 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604763 2573 flags.go:64] FLAG: --log-flush-frequency="5s" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604766 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" May 26 16:00:51.608444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604772 2573 flags.go:64] FLAG: --log-json-split-stream="false" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604775 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604778 2573 flags.go:64] FLAG: --log-text-split-stream="false" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604781 2573 flags.go:64] FLAG: --logging-format="text" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604784 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604787 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604790 2573 flags.go:64] FLAG: --manifest-url="" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604793 2573 flags.go:64] FLAG: --manifest-url-header="" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604797 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604801 2573 flags.go:64] FLAG: --max-open-files="1000000" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604805 2573 flags.go:64] FLAG: --max-pods="110" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604808 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604812 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604815 2573 flags.go:64] FLAG: --memory-manager-policy="None" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604818 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604824 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604842 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604848 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604856 2573 flags.go:64] FLAG: --node-status-max-images="50" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604859 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604862 2573 flags.go:64] FLAG: --oom-score-adj="-999" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604865 2573 flags.go:64] FLAG: --pod-cidr="" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604868 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc" May 26 16:00:51.609030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604874 2573 flags.go:64] FLAG: --pod-manifest-path="" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604877 2573 flags.go:64] FLAG: --pod-max-pids="-1" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604880 2573 flags.go:64] FLAG: --pods-per-core="0" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604883 2573 flags.go:64] FLAG: --port="10250" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604886 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604889 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0be8088c233027691" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604893 2573 flags.go:64] FLAG: --qos-reserved="" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604896 2573 flags.go:64] FLAG: --read-only-port="10255" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604899 2573 flags.go:64] FLAG: --register-node="true" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604902 2573 flags.go:64] FLAG: --register-schedulable="true" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604905 2573 flags.go:64] FLAG: --register-with-taints="" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604909 2573 flags.go:64] FLAG: --registry-burst="10" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604912 2573 flags.go:64] FLAG: --registry-qps="5" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604914 2573 flags.go:64] FLAG: --reserved-cpus="" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604917 2573 flags.go:64] FLAG: --reserved-memory="" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604921 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604924 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604928 2573 flags.go:64] FLAG: --rotate-certificates="false" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604930 2573 flags.go:64] FLAG: --rotate-server-certificates="false" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604933 2573 flags.go:64] FLAG: --runonce="false" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604936 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604939 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604942 2573 flags.go:64] FLAG: --seccomp-default="false" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604945 2573 flags.go:64] FLAG: --serialize-image-pulls="true" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604950 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604953 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" May 26 16:00:51.609597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604956 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604961 2573 flags.go:64] FLAG: --storage-driver-password="root" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604963 2573 flags.go:64] FLAG: --storage-driver-secure="false" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604966 2573 flags.go:64] FLAG: --storage-driver-table="stats" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604969 2573 flags.go:64] FLAG: --storage-driver-user="root" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604972 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604976 2573 flags.go:64] FLAG: --sync-frequency="1m0s" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604979 2573 flags.go:64] FLAG: --system-cgroups="" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604981 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604987 2573 flags.go:64] FLAG: --system-reserved-cgroup="" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604990 2573 flags.go:64] FLAG: --tls-cert-file="" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604993 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.604998 2573 flags.go:64] FLAG: --tls-min-version="" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.605001 2573 flags.go:64] FLAG: --tls-private-key-file="" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.605004 2573 flags.go:64] FLAG: --topology-manager-policy="none" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.605007 2573 flags.go:64] FLAG: --topology-manager-policy-options="" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.605010 2573 flags.go:64] FLAG: --topology-manager-scope="container" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.605013 2573 flags.go:64] FLAG: --v="2" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.605017 2573 flags.go:64] FLAG: --version="false" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.605022 2573 flags.go:64] FLAG: --vmodule="" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.605027 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.605030 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605126 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605129 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:51.610228 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605132 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605135 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605139 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605141 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605144 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605147 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605152 2573 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605155 2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605159 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605163 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605166 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605169 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605171 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605174 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605177 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605179 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605182 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605185 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605187 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605190 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:51.610795 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605192 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605195 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605197 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605200 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605203 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605205 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605208 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605211 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605214 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605216 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605219 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605221 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605224 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605227 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605229 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605232 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605234 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605237 2573 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605240 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605243 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:51.611289 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605246 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605249 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605252 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605254 2573 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605257 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605259 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605262 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605265 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605267 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605270 2573 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605272 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605275 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605277 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605280 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605282 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605285 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605287 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605290 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605292 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605295 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:51.611792 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605297 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605300 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605302 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605304 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605307 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605310 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605312 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605314 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605317 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605320 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605323 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605325 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605328 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605331 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605334 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605337 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605341 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605344 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605347 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:51.612341 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605349 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:51.612842 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605352 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:51.612842 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605356 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:51.612842 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605359 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:51.612842 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.605362 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:51.612842 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.606130 2573 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 26 16:00:51.612842 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.612815 2573 server.go:530] "Kubelet version" kubeletVersion="v1.33.11" May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.612849 2573 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612901 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612907 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612911 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612914 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612917 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612920 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612923 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612925 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612928 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612931 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612934 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612936 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612939 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612941 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612944 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612947 2573 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612950 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612953 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:51.613000 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612956 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612958 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612961 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612964 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612966 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612969 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612971 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612974 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612977 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612981 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612986 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612989 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612992 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612995 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.612999 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613002 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613004 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613007 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613010 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:51.613472 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613013 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613016 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613019 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613021 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613024 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613026 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613029 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613031 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613034 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613037 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613039 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613042 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613044 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613047 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613049 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613052 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613055 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613058 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613061 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:51.613955 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613065 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613068 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613071 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613074 2573 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613076 2573 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613079 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613081 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613084 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613087 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613091 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613094 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613096 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613099 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613102 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613104 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613107 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613109 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613112 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613114 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613117 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:51.614405 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613120 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613122 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613125 2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613127 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613130 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613132 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613135 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613138 2573 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613140 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613142 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.613147 2573 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613271 2573 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613277 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613280 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613283 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 26 16:00:51.614902 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613285 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613288 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613291 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613293 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613296 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613299 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613302 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613305 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613308 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613311 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613313 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613316 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613319 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613321 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613324 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613326 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613329 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613332 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613334 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613337 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 26 16:00:51.615285 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613339 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613342 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613344 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613347 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613350 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613352 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613355 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613357 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613360 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613362 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613365 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613369 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613372 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613375 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613378 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613380 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613382 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613386 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613389 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 26 16:00:51.615781 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613392 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613394 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613397 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613399 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613402 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613404 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613407 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613409 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613412 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613414 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613417 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613419 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613422 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613424 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613427 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613429 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613432 2573 feature_gate.go:328] unrecognized feature gate: NewOLM May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613434 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613437 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613439 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 26 16:00:51.616328 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613442 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613444 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613447 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613449 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613452 2573 feature_gate.go:328] unrecognized feature gate: DualReplica May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613454 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613456 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613459 2573 feature_gate.go:328] unrecognized feature gate: Example May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613461 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613464 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613468 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613471 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613473 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613476 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613479 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613483 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613487 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613490 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613493 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613495 2573 feature_gate.go:328] unrecognized feature gate: Example2 May 26 16:00:51.616802 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613498 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 26 16:00:51.617290 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613501 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 26 16:00:51.617290 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:51.613504 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 26 16:00:51.617290 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.613508 2573 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 26 16:00:51.617290 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.614404 2573 server.go:962] "Client rotation is on, will bootstrap in background" May 26 16:00:51.617290 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.616668 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" May 26 16:00:51.617748 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.617736 2573 server.go:1019] "Starting client certificate rotation" May 26 16:00:51.617862 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.617846 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" May 26 16:00:51.618751 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.618740 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" May 26 16:00:51.647759 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.647742 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 26 16:00:51.652361 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.652320 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 26 16:00:51.665802 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.665779 2573 log.go:25] "Validated CRI v1 runtime API" May 26 16:00:51.674029 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.674012 2573 log.go:25] "Validated CRI v1 image API" May 26 16:00:51.679233 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.679169 2573 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" May 26 16:00:51.682673 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.682647 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 96361b93-0900-4623-b984-bfa9715d9798:/dev/nvme0n1p4 e26d4563-3e7a-4c20-960a-e65510adebdf:/dev/nvme0n1p3] May 26 16:00:51.682751 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.682669 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] May 26 16:00:51.687044 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.686934 2573 manager.go:217] Machine: {Timestamp:2026-05-26 16:00:51.686400844 +0000 UTC m=+0.546323304 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095327 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f49d7638a0937753cbf38a3a0daa2 SystemUUID:ec2f49d7-638a-0937-753c-bf38a3a0daa2 BootID:eea12dc7-9f49-4616-8263-5a3f09328600 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f5:d7:29:5a:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f5:d7:29:5a:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:68:23:b9:47:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} May 26 16:00:51.687044 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.687040 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. May 26 16:00:51.687157 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.687120 2573 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} May 26 16:00:51.689711 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.689686 2573 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] May 26 16:00:51.689858 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.689710 2573 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-48.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} May 26 16:00:51.690567 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.690557 2573 topology_manager.go:138] "Creating topology manager with none policy" May 26 16:00:51.690602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.690570 2573 container_manager_linux.go:306] "Creating device plugin manager" May 26 16:00:51.690602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.690577 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" May 26 16:00:51.691462 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.691450 2573 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" May 26 16:00:51.694011 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.694000 2573 state_mem.go:36] "Initialized new in-memory state store" May 26 16:00:51.694192 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.694180 2573 server.go:1267] "Using root directory" path="/var/lib/kubelet" May 26 16:00:51.696211 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.696196 2573 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 26 16:00:51.697024 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.697013 2573 kubelet.go:491] "Attempting to sync node with API server" May 26 16:00:51.697060 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.697030 2573 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" May 26 16:00:51.697060 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.697042 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" May 26 16:00:51.697060 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.697050 2573 kubelet.go:397] "Adding apiserver pod source" May 26 16:00:51.697060 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.697058 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" May 26 16:00:51.698616 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.698601 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 26 16:00:51.698656 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.698629 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 26 16:00:51.703506 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.703487 2573 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1" May 26 16:00:51.707626 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.707605 2573 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" May 26 16:00:51.708639 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708628 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" May 26 16:00:51.708685 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708645 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" May 26 16:00:51.708685 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708653 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" May 26 16:00:51.708685 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708659 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" May 26 16:00:51.708685 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708669 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" May 26 16:00:51.708685 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708678 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" May 26 16:00:51.708685 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708686 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" May 26 16:00:51.708862 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708692 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" May 26 16:00:51.708862 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708699 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" May 26 16:00:51.708862 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708705 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" May 26 16:00:51.708862 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.708714 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" May 26 16:00:51.709100 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.709082 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" May 26 16:00:51.710933 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.710920 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" May 26 16:00:51.710933 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.710933 2573 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" May 26 16:00:51.712955 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.712935 2573 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" May 26 16:00:51.713025 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.712936 2573 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" May 26 16:00:51.714213 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.714201 2573 watchdog_linux.go:99] "Systemd watchdog is not enabled" May 26 16:00:51.714250 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.714238 2573 server.go:1295] "Started kubelet" May 26 16:00:51.714343 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.714316 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 May 26 16:00:51.714391 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.714328 2573 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 May 26 16:00:51.714444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.714399 2573 server_v1.go:47] "podresources" method="list" useActivePods=true May 26 16:00:51.715045 ip-10-0-140-48 systemd[1]: Started Kubernetes Kubelet. May 26 16:00:51.715689 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.715616 2573 server.go:317] "Adding debug handlers to kubelet server" May 26 16:00:51.716010 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.715905 2573 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" May 26 16:00:51.724551 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.724527 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" May 26 16:00:51.724941 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.724904 2573 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" May 26 16:00:51.725114 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.725099 2573 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope May 26 16:00:51.725183 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.725164 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" May 26 16:00:51.725371 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.725354 2573 volume_manager.go:295] "The desired_state_of_world populator starts" May 26 16:00:51.725452 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.725374 2573 volume_manager.go:297] "Starting Kubelet Volume Manager" May 26 16:00:51.725495 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.725477 2573 desired_state_of_world_populator.go:150] "Desired state populator starts to run" May 26 16:00:51.725538 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.725524 2573 reconstruct.go:97] "Volume reconstruction finished" May 26 16:00:51.725538 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.725531 2573 reconciler.go:26] "Reconciler: start to sync state" May 26 16:00:51.725999 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.725983 2573 factory.go:55] Registering systemd factory May 26 16:00:51.726092 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.726083 2573 factory.go:223] Registration of the systemd container factory successfully May 26 16:00:51.726408 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.724954 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-48.ec2.internal.18b328e751ab4725 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-48.ec2.internal,UID:ip-10-0-140-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-48.ec2.internal,},FirstTimestamp:2026-05-26 16:00:51.714213669 +0000 UTC m=+0.574136132,LastTimestamp:2026-05-26 16:00:51.714213669 +0000 UTC m=+0.574136132,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-48.ec2.internal,}" May 26 16:00:51.726492 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.726400 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:51.726779 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.726762 2573 factory.go:153] Registering CRI-O factory May 26 16:00:51.726779 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.726781 2573 factory.go:223] Registration of the crio container factory successfully May 26 16:00:51.726954 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.726870 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory May 26 16:00:51.726954 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.726889 2573 factory.go:103] Registering Raw factory May 26 16:00:51.726954 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.726905 2573 manager.go:1196] Started watching for new ooms in manager May 26 16:00:51.727279 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.727266 2573 manager.go:319] Starting recovery of all containers May 26 16:00:51.727493 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.727467 2573 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" May 26 16:00:51.727493 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.727474 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" May 26 16:00:51.734801 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.734594 2573 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" May 26 16:00:51.739683 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.739666 2573 manager.go:324] Recovery completed May 26 16:00:51.743818 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.743803 2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:51.746763 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.746734 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:51.746817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.746777 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:51.746817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.746788 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:51.747307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.747294 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" May 26 16:00:51.747366 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.747308 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" May 26 16:00:51.747366 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.747328 2573 state_mem.go:36] "Initialized new in-memory state store" May 26 16:00:51.751033 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.751020 2573 policy_none.go:49] "None policy: Start" May 26 16:00:51.751075 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.751037 2573 memory_manager.go:186] "Starting memorymanager" policy="None" May 26 16:00:51.751075 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.751052 2573 state_mem.go:35] "Initializing new in-memory state store" May 26 16:00:51.764903 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.764821 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-48.ec2.internal.18b328e7539bf4ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-48.ec2.internal,UID:ip-10-0-140-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-48.ec2.internal,},FirstTimestamp:2026-05-26 16:00:51.746763948 +0000 UTC m=+0.606686412,LastTimestamp:2026-05-26 16:00:51.746763948 +0000 UTC m=+0.606686412,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-48.ec2.internal,}" May 26 16:00:51.779097 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.779032 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-48.ec2.internal.18b328e7539c3d01 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-48.ec2.internal,UID:ip-10-0-140-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-48.ec2.internal,},FirstTimestamp:2026-05-26 16:00:51.746782465 +0000 UTC m=+0.606704924,LastTimestamp:2026-05-26 16:00:51.746782465 +0000 UTC m=+0.606704924,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-48.ec2.internal,}" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.789924 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-48.ec2.internal.18b328e7539c60e0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-48.ec2.internal,UID:ip-10-0-140-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-48.ec2.internal,},FirstTimestamp:2026-05-26 16:00:51.746791648 +0000 UTC m=+0.606714108,LastTimestamp:2026-05-26 16:00:51.746791648 +0000 UTC m=+0.606714108,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-48.ec2.internal,}" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.790779 2573 manager.go:341] "Starting Device Plugin manager" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.790807 2573 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.790817 2573 server.go:85] "Starting device plugin registration server" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.791082 2573 eviction_manager.go:189] "Eviction manager: starting control loop" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.791096 2573 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.791183 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.791260 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.791269 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.791896 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.791937 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.796119 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4fjqr" May 26 16:00:51.817293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.805387 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4fjqr" May 26 16:00:51.868711 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.868684 2573 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" May 26 16:00:51.868711 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.868709 2573 status_manager.go:230] "Starting to sync pod status with apiserver" May 26 16:00:51.868911 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.868730 2573 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." May 26 16:00:51.868911 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.868737 2573 kubelet.go:2451] "Starting kubelet main sync loop" May 26 16:00:51.868911 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.868774 2573 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" May 26 16:00:51.871645 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.871627 2573 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:51.891875 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.891861 2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:51.892878 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.892861 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:51.892952 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.892890 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:51.892952 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.892901 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:51.892952 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.892925 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-48.ec2.internal" May 26 16:00:51.902874 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.902860 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-48.ec2.internal" May 26 16:00:51.902935 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.902880 2573 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-48.ec2.internal\": node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:51.922507 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.922487 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:51.969815 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.969728 2573 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal"] May 26 16:00:51.969979 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.969844 2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:51.970850 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.970819 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:51.970910 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.970868 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:51.970910 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.970889 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:51.971957 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.971945 2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:51.972184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.972171 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" May 26 16:00:51.972225 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.972200 2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:51.974074 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.974053 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:51.974207 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.974087 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:51.974207 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.974092 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:51.974207 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.974101 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:51.974207 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.974110 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:51.974207 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.974122 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:51.975446 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.975419 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal" May 26 16:00:51.975446 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.975450 2573 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 26 16:00:51.977955 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.976931 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientMemory" May 26 16:00:51.977955 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.976960 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasNoDiskPressure" May 26 16:00:51.977955 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:51.976990 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeHasSufficientPID" May 26 16:00:51.993351 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.993328 2573 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-48.ec2.internal\" not found" node="ip-10-0-140-48.ec2.internal" May 26 16:00:51.997069 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:51.997052 2573 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-48.ec2.internal\" not found" node="ip-10-0-140-48.ec2.internal" May 26 16:00:52.022644 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.022611 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:52.026946 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.026924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b8861c8ce95eee7825ce03628bd1faf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal\" (UID: \"1b8861c8ce95eee7825ce03628bd1faf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" May 26 16:00:52.027024 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.026955 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b8861c8ce95eee7825ce03628bd1faf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal\" (UID: \"1b8861c8ce95eee7825ce03628bd1faf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" May 26 16:00:52.027024 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.026981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5f6d5d2a73b24fd057a0b80db1b7899f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-48.ec2.internal\" (UID: \"5f6d5d2a73b24fd057a0b80db1b7899f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal" May 26 16:00:52.123373 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.123349 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:52.127634 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.127621 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b8861c8ce95eee7825ce03628bd1faf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal\" (UID: \"1b8861c8ce95eee7825ce03628bd1faf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" May 26 16:00:52.127690 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.127643 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b8861c8ce95eee7825ce03628bd1faf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal\" (UID: \"1b8861c8ce95eee7825ce03628bd1faf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" May 26 16:00:52.127690 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.127659 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5f6d5d2a73b24fd057a0b80db1b7899f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-48.ec2.internal\" (UID: \"5f6d5d2a73b24fd057a0b80db1b7899f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal" May 26 16:00:52.127754 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.127700 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5f6d5d2a73b24fd057a0b80db1b7899f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-48.ec2.internal\" (UID: \"5f6d5d2a73b24fd057a0b80db1b7899f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal" May 26 16:00:52.127754 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.127724 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b8861c8ce95eee7825ce03628bd1faf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal\" (UID: \"1b8861c8ce95eee7825ce03628bd1faf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" May 26 16:00:52.127754 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.127734 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b8861c8ce95eee7825ce03628bd1faf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal\" (UID: \"1b8861c8ce95eee7825ce03628bd1faf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" May 26 16:00:52.223959 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.223880 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:52.296362 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.296339 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" May 26 16:00:52.299082 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.299064 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal" May 26 16:00:52.324557 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.324534 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:52.424997 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.424961 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:52.525428 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.525402 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:52.617957 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.617927 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" May 26 16:00:52.618486 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.618068 2573 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 26 16:00:52.626094 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.626065 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:52.724891 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.724856 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" May 26 16:00:52.726292 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.726269 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:52.742709 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.742682 2573 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 26 16:00:52.769892 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.769864 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ckp2g" May 26 16:00:52.775683 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.775659 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ckp2g" May 26 16:00:52.788773 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:52.788746 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8861c8ce95eee7825ce03628bd1faf.slice/crio-634a1e0bdf50b17c2f4186d7b841e980b41d100f169d865701ff2b4cdbd88691 WatchSource:0}: Error finding container 634a1e0bdf50b17c2f4186d7b841e980b41d100f169d865701ff2b4cdbd88691: Status 404 returned error can't find the container with id 634a1e0bdf50b17c2f4186d7b841e980b41d100f169d865701ff2b4cdbd88691 May 26 16:00:52.788929 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:52.788913 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6d5d2a73b24fd057a0b80db1b7899f.slice/crio-11988765823af0acca9c1773ea5336767a58bd47966d9c84e2c51c4d5a7c7006 WatchSource:0}: Error finding container 11988765823af0acca9c1773ea5336767a58bd47966d9c84e2c51c4d5a7c7006: Status 404 returned error can't find the container with id 11988765823af0acca9c1773ea5336767a58bd47966d9c84e2c51c4d5a7c7006 May 26 16:00:52.794412 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.794396 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:00:52.807888 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.807861 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 15:55:51 +0000 UTC" deadline="2027-11-03 00:37:48.046588554 +0000 UTC" May 26 16:00:52.807888 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.807886 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12608h36m55.238705442s" May 26 16:00:52.827116 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.827092 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:52.853836 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.853816 2573 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:52.871468 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.871415 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal" event={"ID":"5f6d5d2a73b24fd057a0b80db1b7899f","Type":"ContainerStarted","Data":"11988765823af0acca9c1773ea5336767a58bd47966d9c84e2c51c4d5a7c7006"} May 26 16:00:52.872356 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:52.872328 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" event={"ID":"1b8861c8ce95eee7825ce03628bd1faf","Type":"ContainerStarted","Data":"634a1e0bdf50b17c2f4186d7b841e980b41d100f169d865701ff2b4cdbd88691"} May 26 16:00:52.927711 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:52.927690 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:53.028237 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.028167 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:53.067806 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.067781 2573 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:53.128568 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.128531 2573 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-48.ec2.internal\" not found" May 26 16:00:53.135345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.135320 2573 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:53.225573 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.225540 2573 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" May 26 16:00:53.243941 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.243900 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 26 16:00:53.245196 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.245175 2573 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal" May 26 16:00:53.257695 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.257598 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 26 16:00:53.698509 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.698477 2573 apiserver.go:52] "Watching apiserver" May 26 16:00:53.707447 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.707413 2573 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" May 26 16:00:53.707868 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.707845 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m","openshift-image-registry/node-ca-b4gnl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal","openshift-multus/network-metrics-daemon-vz84d","openshift-network-diagnostics/network-check-target-sdx75","openshift-ovn-kubernetes/ovnkube-node-9pgwm","kube-system/konnectivity-agent-7m96n","openshift-cluster-node-tuning-operator/tuned-8xgrm","openshift-dns/node-resolver-vjm54","openshift-multus/multus-additional-cni-plugins-r4lmb","openshift-multus/multus-vn7x4","openshift-network-operator/iptables-alerter-bqqjm"] May 26 16:00:53.710629 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.710607 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.712684 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.712664 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.712956 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.712936 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" May 26 16:00:53.713303 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.713271 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" May 26 16:00:53.713386 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.713337 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p7mrf\"" May 26 16:00:53.714684 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.714652 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.715890 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.715360 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" May 26 16:00:53.715890 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.715691 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" May 26 16:00:53.715890 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.715728 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" May 26 16:00:53.715890 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.715808 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gmxkn\"" May 26 16:00:53.717914 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.717458 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fdt45\"" May 26 16:00:53.717914 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.717822 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" May 26 16:00:53.717914 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.717888 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" May 26 16:00:53.718176 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.718154 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" May 26 16:00:53.719508 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.719243 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vn7x4" May 26 16:00:53.721100 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.721079 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:53.721189 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.721167 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:00:53.723803 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.723398 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" May 26 16:00:53.723803 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.723475 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bnzht\"" May 26 16:00:53.723803 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.723710 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" May 26 16:00:53.724077 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.723950 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" May 26 16:00:53.724077 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.724062 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" May 26 16:00:53.724461 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.724442 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.726587 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.726565 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.727554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.727443 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" May 26 16:00:53.727554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.727476 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jczks\"" May 26 16:00:53.727554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.727493 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" May 26 16:00:53.728695 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.728678 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" May 26 16:00:53.728792 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.728698 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" May 26 16:00:53.729445 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.729049 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9c79z\"" May 26 16:00:53.729445 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.729107 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7m96n" May 26 16:00:53.731633 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.731228 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" May 26 16:00:53.731633 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.731276 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" May 26 16:00:53.731633 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.731508 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gd4dh\"" May 26 16:00:53.731817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.731667 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.733754 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.733736 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:00:53.733861 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.733801 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:00:53.734095 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.734073 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" May 26 16:00:53.735003 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.734970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-modprobe-d\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.735104 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2367058a-a723-4dcc-a10d-f33b1670f57e-tmp\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.735104 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.735104 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a-host\") pod \"node-ca-b4gnl\" (UID: \"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a\") " pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.735104 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-registration-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.735104 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-var-lib-kubelet\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-run\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-sys-fs\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-cnibin\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-run-k8s-cni-cncf-io\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-etc-kubernetes\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735226 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-host\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-daemon-config\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735247 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rgtn7\"" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735256 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e636febb-62af-4bae-87f2-f5ba6ee79f71-hosts-file\") pod \"node-resolver-vjm54\" (UID: \"e636febb-62af-4bae-87f2-f5ba6ee79f71\") " pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-sysctl-conf\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-sys\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-run-netns\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.735345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-sysctl-d\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-tuned\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-cni-dir\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-hostroot\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56pm\" (UniqueName: \"kubernetes.io/projected/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-kube-api-access-c56pm\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-cni-binary-copy\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735468 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e636febb-62af-4bae-87f2-f5ba6ee79f71-tmp-dir\") pod \"node-resolver-vjm54\" (UID: \"e636febb-62af-4bae-87f2-f5ba6ee79f71\") " pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmg6\" (UniqueName: \"kubernetes.io/projected/2367058a-a723-4dcc-a10d-f33b1670f57e-kube-api-access-qxmg6\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735494 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735503 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-socket-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-conf-dir\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735534 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735577 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735652 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735497 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-system-cni-dir\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-lib-modules\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fee4bfe-ae19-487c-a75b-cf4f2a013075-cni-binary-copy\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.736006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.735955 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fee4bfe-ae19-487c-a75b-cf4f2a013075-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkh4\" (UniqueName: \"kubernetes.io/projected/bc05844d-f6f9-4cdd-a748-a601268f1ae1-kube-api-access-cpkh4\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75ph7\" (UniqueName: \"kubernetes.io/projected/5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a-kube-api-access-75ph7\") pod \"node-ca-b4gnl\" (UID: \"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a\") " pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-socket-dir-parent\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-os-release\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736216 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736269 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-run-multus-certs\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-system-cni-dir\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-var-lib-kubelet\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-cnibin\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-systemd\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736398 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-var-lib-cni-bin\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736423 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-var-lib-cni-multus\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fee4bfe-ae19-487c-a75b-cf4f2a013075-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgt6h\" (UniqueName: \"kubernetes.io/projected/e636febb-62af-4bae-87f2-f5ba6ee79f71-kube-api-access-hgt6h\") pod \"node-resolver-vjm54\" (UID: \"e636febb-62af-4bae-87f2-f5ba6ee79f71\") " pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736500 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-sysconfig\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.736886 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-kubernetes\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.737606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-device-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.737606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736585 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a-serviceca\") pod \"node-ca-b4gnl\" (UID: \"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a\") " pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.737606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736619 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-os-release\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.737606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736664 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6prts\" (UniqueName: \"kubernetes.io/projected/264cf2a2-6d0f-4220-8797-88813b9a33fe-kube-api-access-6prts\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:53.737606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.736705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7bg\" (UniqueName: \"kubernetes.io/projected/5fee4bfe-ae19-487c-a75b-cf4f2a013075-kube-api-access-fr7bg\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.738221 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.738201 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.740630 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.740614 2573 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 26 16:00:53.742004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.741741 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vczzv\"" May 26 16:00:53.742004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.741789 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" May 26 16:00:53.742004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.741808 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" May 26 16:00:53.742004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.741749 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" May 26 16:00:53.776424 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.776391 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:52 +0000 UTC" deadline="2027-12-14 23:29:33.82505431 +0000 UTC" May 26 16:00:53.776424 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.776423 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13615h28m40.048634473s" May 26 16:00:53.826567 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.826537 2573 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" May 26 16:00:53.837115 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837080 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-lib-modules\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.837258 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837124 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fee4bfe-ae19-487c-a75b-cf4f2a013075-cni-binary-copy\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.837258 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837147 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fee4bfe-ae19-487c-a75b-cf4f2a013075-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.837258 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837166 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkh4\" (UniqueName: \"kubernetes.io/projected/bc05844d-f6f9-4cdd-a748-a601268f1ae1-kube-api-access-cpkh4\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.837411 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837260 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-lib-modules\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.837411 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837285 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-75ph7\" (UniqueName: \"kubernetes.io/projected/5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a-kube-api-access-75ph7\") pod \"node-ca-b4gnl\" (UID: \"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a\") " pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.837411 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837323 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-socket-dir-parent\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.837411 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837348 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-os-release\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.837411 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-kubelet\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.837411 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837403 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837429 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-run-multus-certs\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-slash\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837460 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-socket-dir-parent\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837482 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-system-cni-dir\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837506 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-os-release\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837526 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837553 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-system-cni-dir\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837560 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-var-lib-kubelet\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837595 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-var-lib-kubelet\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837596 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-run-multus-certs\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837599 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-cnibin\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837628 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-cnibin\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-log-socket\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.837678 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837675 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-systemd\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837719 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-systemd\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837719 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fee4bfe-ae19-487c-a75b-cf4f2a013075-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837745 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-var-lib-cni-bin\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837782 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-var-lib-cni-multus\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837719 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fee4bfe-ae19-487c-a75b-cf4f2a013075-cni-binary-copy\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837805 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fee4bfe-ae19-487c-a75b-cf4f2a013075-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837855 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hgt6h\" (UniqueName: \"kubernetes.io/projected/e636febb-62af-4bae-87f2-f5ba6ee79f71-kube-api-access-hgt6h\") pod \"node-resolver-vjm54\" (UID: \"e636febb-62af-4bae-87f2-f5ba6ee79f71\") " pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837886 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-etc-openvswitch\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837920 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.837980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-var-lib-cni-bin\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838024 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8-iptables-alerter-script\") pod \"iptables-alerter-bqqjm\" (UID: \"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8\") " pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838058 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-sysconfig\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838098 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-kubernetes\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-sysconfig\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838128 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-device-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838152 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a-serviceca\") pod \"node-ca-b4gnl\" (UID: \"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a\") " pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.838335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838138 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-var-lib-cni-multus\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838166 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-kubernetes\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838180 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-os-release\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-device-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838226 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6prts\" (UniqueName: \"kubernetes.io/projected/264cf2a2-6d0f-4220-8797-88813b9a33fe-kube-api-access-6prts\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838243 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-os-release\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838268 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7bg\" (UniqueName: \"kubernetes.io/projected/5fee4bfe-ae19-487c-a75b-cf4f2a013075-kube-api-access-fr7bg\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-node-log\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838331 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-modprobe-d\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838359 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2367058a-a723-4dcc-a10d-f33b1670f57e-tmp\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838383 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838387 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fee4bfe-ae19-487c-a75b-cf4f2a013075-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838407 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a-host\") pod \"node-ca-b4gnl\" (UID: \"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a\") " pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8-host-slash\") pod \"iptables-alerter-bqqjm\" (UID: \"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8\") " pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838486 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-registration-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838510 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.839094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838536 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-run-netns\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838539 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-modprobe-d\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-run-openvswitch\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838597 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a-serviceca\") pod \"node-ca-b4gnl\" (UID: \"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a\") " pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838643 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838654 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838655 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a-host\") pod \"node-ca-b4gnl\" (UID: \"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a\") " pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838682 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838692 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-var-lib-kubelet\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838699 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b420341-5a85-4184-9d56-616ba0fa532c-env-overrides\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838724 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-registration-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838741 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-var-lib-kubelet\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838764 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b420341-5a85-4184-9d56-616ba0fa532c-ovn-node-metrics-cert\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838793 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-run\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838819 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-sys-fs\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838861 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-cnibin\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838887 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-run\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.839817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838936 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-run-k8s-cni-cncf-io\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838948 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-cnibin\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838949 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-sys-fs\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838981 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-etc-kubernetes\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/616802f2-6a23-4737-921e-862e05ea782c-konnectivity-ca\") pod \"konnectivity-agent-7m96n\" (UID: \"616802f2-6a23-4737-921e-862e05ea782c\") " pod="kube-system/konnectivity-agent-7m96n" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839026 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-etc-kubernetes\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839037 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-host\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.838982 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-run-k8s-cni-cncf-io\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839072 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-daemon-config\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839081 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-host\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839102 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e636febb-62af-4bae-87f2-f5ba6ee79f71-hosts-file\") pod \"node-resolver-vjm54\" (UID: \"e636febb-62af-4bae-87f2-f5ba6ee79f71\") " pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-run-ovn\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b420341-5a85-4184-9d56-616ba0fa532c-ovnkube-config\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b420341-5a85-4184-9d56-616ba0fa532c-ovnkube-script-lib\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839233 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e636febb-62af-4bae-87f2-f5ba6ee79f71-hosts-file\") pod \"node-resolver-vjm54\" (UID: \"e636febb-62af-4bae-87f2-f5ba6ee79f71\") " pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839234 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxqt\" (UniqueName: \"kubernetes.io/projected/37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8-kube-api-access-bfxqt\") pod \"iptables-alerter-bqqjm\" (UID: \"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8\") " pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839276 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-sysctl-conf\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839295 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-sys\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.840615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839319 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-run-netns\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839357 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-host-run-netns\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839373 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-sys\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839419 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839457 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xj6\" (UniqueName: \"kubernetes.io/projected/5b420341-5a85-4184-9d56-616ba0fa532c-kube-api-access-t5xj6\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839485 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-sysctl-d\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.839507 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839507 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-tuned\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839546 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-cni-dir\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.839601 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs podName:264cf2a2-6d0f-4220-8797-88813b9a33fe nodeName:}" failed. No retries permitted until 2026-05-26 16:00:54.339556711 +0000 UTC m=+3.199479171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs") pod "network-metrics-daemon-vz84d" (UID: "264cf2a2-6d0f-4220-8797-88813b9a33fe") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839421 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-sysctl-conf\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839611 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-sysctl-d\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839618 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-cni-dir\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839620 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-hostroot\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839651 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-hostroot\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839597 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-daemon-config\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839662 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c56pm\" (UniqueName: \"kubernetes.io/projected/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-kube-api-access-c56pm\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.841406 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839702 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-systemd-units\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-run-systemd\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-var-lib-openvswitch\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839801 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-cni-binary-copy\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839853 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e636febb-62af-4bae-87f2-f5ba6ee79f71-tmp-dir\") pod \"node-resolver-vjm54\" (UID: \"e636febb-62af-4bae-87f2-f5ba6ee79f71\") " pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-cni-bin\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/616802f2-6a23-4737-921e-862e05ea782c-agent-certs\") pod \"konnectivity-agent-7m96n\" (UID: \"616802f2-6a23-4737-921e-862e05ea782c\") " pod="kube-system/konnectivity-agent-7m96n" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839928 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmg6\" (UniqueName: \"kubernetes.io/projected/2367058a-a723-4dcc-a10d-f33b1670f57e-kube-api-access-qxmg6\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839968 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-socket-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.839993 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-conf-dir\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.840018 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-system-cni-dir\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.840045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-cni-netd\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.840094 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e636febb-62af-4bae-87f2-f5ba6ee79f71-tmp-dir\") pod \"node-resolver-vjm54\" (UID: \"e636febb-62af-4bae-87f2-f5ba6ee79f71\") " pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.840186 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-multus-conf-dir\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.840214 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fee4bfe-ae19-487c-a75b-cf4f2a013075-system-cni-dir\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.840258 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-cni-binary-copy\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.840292 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc05844d-f6f9-4cdd-a748-a601268f1ae1-socket-dir\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.842193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.842157 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2367058a-a723-4dcc-a10d-f33b1670f57e-etc-tuned\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.842643 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.842240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2367058a-a723-4dcc-a10d-f33b1670f57e-tmp\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.849371 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.849352 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkh4\" (UniqueName: \"kubernetes.io/projected/bc05844d-f6f9-4cdd-a748-a601268f1ae1-kube-api-access-cpkh4\") pod \"aws-ebs-csi-driver-node-qvp5m\" (UID: \"bc05844d-f6f9-4cdd-a748-a601268f1ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:53.850940 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.850917 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75ph7\" (UniqueName: \"kubernetes.io/projected/5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a-kube-api-access-75ph7\") pod \"node-ca-b4gnl\" (UID: \"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a\") " pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:53.856712 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.856690 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmg6\" (UniqueName: \"kubernetes.io/projected/2367058a-a723-4dcc-a10d-f33b1670f57e-kube-api-access-qxmg6\") pod \"tuned-8xgrm\" (UID: \"2367058a-a723-4dcc-a10d-f33b1670f57e\") " pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:53.858076 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.858055 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6prts\" (UniqueName: \"kubernetes.io/projected/264cf2a2-6d0f-4220-8797-88813b9a33fe-kube-api-access-6prts\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:53.858983 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.858964 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgt6h\" (UniqueName: \"kubernetes.io/projected/e636febb-62af-4bae-87f2-f5ba6ee79f71-kube-api-access-hgt6h\") pod \"node-resolver-vjm54\" (UID: \"e636febb-62af-4bae-87f2-f5ba6ee79f71\") " pod="openshift-dns/node-resolver-vjm54" May 26 16:00:53.860038 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.860021 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56pm\" (UniqueName: \"kubernetes.io/projected/7f746638-ec1d-4cc7-8e72-6f7ef6991c06-kube-api-access-c56pm\") pod \"multus-vn7x4\" (UID: \"7f746638-ec1d-4cc7-8e72-6f7ef6991c06\") " pod="openshift-multus/multus-vn7x4" May 26 16:00:53.863265 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.863243 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7bg\" (UniqueName: \"kubernetes.io/projected/5fee4bfe-ae19-487c-a75b-cf4f2a013075-kube-api-access-fr7bg\") pod \"multus-additional-cni-plugins-r4lmb\" (UID: \"5fee4bfe-ae19-487c-a75b-cf4f2a013075\") " pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:53.932116 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.932036 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-gjfr7"] May 26 16:00:53.938210 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.938179 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:53.938345 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.938247 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:00:53.940267 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940245 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/616802f2-6a23-4737-921e-862e05ea782c-konnectivity-ca\") pod \"konnectivity-agent-7m96n\" (UID: \"616802f2-6a23-4737-921e-862e05ea782c\") " pod="kube-system/konnectivity-agent-7m96n" May 26 16:00:53.940383 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940273 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-run-ovn\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940383 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940292 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b420341-5a85-4184-9d56-616ba0fa532c-ovnkube-config\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940383 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940315 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b420341-5a85-4184-9d56-616ba0fa532c-ovnkube-script-lib\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940383 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940338 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxqt\" (UniqueName: \"kubernetes.io/projected/37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8-kube-api-access-bfxqt\") pod \"iptables-alerter-bqqjm\" (UID: \"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8\") " pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.940383 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940367 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-run-ovn\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940559 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940374 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xj6\" (UniqueName: \"kubernetes.io/projected/5b420341-5a85-4184-9d56-616ba0fa532c-kube-api-access-t5xj6\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940728 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940704 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-systemd-units\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940743 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-run-systemd\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940768 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-var-lib-openvswitch\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940791 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-cni-bin\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940794 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-systemd-units\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940810 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-run-systemd\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940814 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/616802f2-6a23-4737-921e-862e05ea782c-agent-certs\") pod \"konnectivity-agent-7m96n\" (UID: \"616802f2-6a23-4737-921e-862e05ea782c\") " pod="kube-system/konnectivity-agent-7m96n" May 26 16:00:53.940915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940870 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-var-lib-openvswitch\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940876 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-cni-netd\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.940915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940913 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-cni-netd\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-cni-bin\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940922 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-kubelet\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940961 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-slash\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/616802f2-6a23-4737-921e-862e05ea782c-konnectivity-ca\") pod \"konnectivity-agent-7m96n\" (UID: \"616802f2-6a23-4737-921e-862e05ea782c\") " pod="kube-system/konnectivity-agent-7m96n" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940991 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-log-socket\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.940963 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-kubelet\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941022 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-etc-openvswitch\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941024 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-slash\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941050 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-log-socket\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941070 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941112 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941117 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-etc-openvswitch\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941147 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8-iptables-alerter-script\") pod \"iptables-alerter-bqqjm\" (UID: \"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8\") " pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941182 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-node-log\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941206 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941223 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8-host-slash\") pod \"iptables-alerter-bqqjm\" (UID: \"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8\") " pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.941349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941226 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-node-log\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941239 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-run-netns\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941255 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-run-openvswitch\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941270 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941277 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941302 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b420341-5a85-4184-9d56-616ba0fa532c-env-overrides\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941312 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-host-run-netns\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941315 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8-host-slash\") pod \"iptables-alerter-bqqjm\" (UID: \"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8\") " pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941326 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b420341-5a85-4184-9d56-616ba0fa532c-ovn-node-metrics-cert\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941469 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b420341-5a85-4184-9d56-616ba0fa532c-run-openvswitch\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941560 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b420341-5a85-4184-9d56-616ba0fa532c-ovnkube-config\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941641 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b420341-5a85-4184-9d56-616ba0fa532c-ovnkube-script-lib\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.941713 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b420341-5a85-4184-9d56-616ba0fa532c-env-overrides\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.942765 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.942363 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8-iptables-alerter-script\") pod \"iptables-alerter-bqqjm\" (UID: \"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8\") " pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.943146 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.943105 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/616802f2-6a23-4737-921e-862e05ea782c-agent-certs\") pod \"konnectivity-agent-7m96n\" (UID: \"616802f2-6a23-4737-921e-862e05ea782c\") " pod="kube-system/konnectivity-agent-7m96n" May 26 16:00:53.943559 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.943539 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b420341-5a85-4184-9d56-616ba0fa532c-ovn-node-metrics-cert\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:53.955415 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.955394 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:00:53.955415 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.955420 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:00:53.955552 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.955433 2573 projected.go:194] Error preparing data for projected volume kube-api-access-tk4bl for pod openshift-network-diagnostics/network-check-target-sdx75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:53.955552 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:53.955513 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl podName:2945e20a-dab7-473f-a55f-396c99f0d855 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:54.455491661 +0000 UTC m=+3.315414114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tk4bl" (UniqueName: "kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl") pod "network-check-target-sdx75" (UID: "2945e20a-dab7-473f-a55f-396c99f0d855") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:53.957329 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.957308 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxqt\" (UniqueName: \"kubernetes.io/projected/37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8-kube-api-access-bfxqt\") pod \"iptables-alerter-bqqjm\" (UID: \"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8\") " pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:53.958588 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:53.958573 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xj6\" (UniqueName: \"kubernetes.io/projected/5b420341-5a85-4184-9d56-616ba0fa532c-kube-api-access-t5xj6\") pod \"ovnkube-node-9pgwm\" (UID: \"5b420341-5a85-4184-9d56-616ba0fa532c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:54.026976 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.026937 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" May 26 16:00:54.034721 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.034692 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" May 26 16:00:54.041611 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.041583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:54.041724 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.041710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89ec24bc-600b-4510-a6e5-95c672ab9577-kubelet-config\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:54.041805 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.041786 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89ec24bc-600b-4510-a6e5-95c672ab9577-dbus\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:54.047337 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.047316 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b4gnl" May 26 16:00:54.050911 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.050890 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vn7x4" May 26 16:00:54.057465 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.057450 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" May 26 16:00:54.065963 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.065948 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjm54" May 26 16:00:54.072491 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.072475 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7m96n" May 26 16:00:54.080117 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.080099 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:00:54.085571 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.085555 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bqqjm" May 26 16:00:54.142454 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.142424 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89ec24bc-600b-4510-a6e5-95c672ab9577-kubelet-config\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:54.142638 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.142481 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89ec24bc-600b-4510-a6e5-95c672ab9577-dbus\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:54.142638 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.142534 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:54.142638 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.142563 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89ec24bc-600b-4510-a6e5-95c672ab9577-kubelet-config\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:54.142794 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.142692 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89ec24bc-600b-4510-a6e5-95c672ab9577-dbus\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:54.142794 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.142703 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:00:54.142794 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.142778 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret podName:89ec24bc-600b-4510-a6e5-95c672ab9577 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:54.642754479 +0000 UTC m=+3.502676945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret") pod "global-pull-secret-syncer-gjfr7" (UID: "89ec24bc-600b-4510-a6e5-95c672ab9577") : object "kube-system"/"original-pull-secret" not registered May 26 16:00:54.314614 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:54.314582 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode636febb_62af_4bae_87f2_f5ba6ee79f71.slice/crio-be74db668bd289d570ca3d98da94e7bf76c20901759a44b298ea849b06db1e95 WatchSource:0}: Error finding container be74db668bd289d570ca3d98da94e7bf76c20901759a44b298ea849b06db1e95: Status 404 returned error can't find the container with id be74db668bd289d570ca3d98da94e7bf76c20901759a44b298ea849b06db1e95 May 26 16:00:54.316479 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:54.316410 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f746638_ec1d_4cc7_8e72_6f7ef6991c06.slice/crio-bc4ab03a71538da7bb0043fc85be137dad06cb0ce19cd764cb98901227494f49 WatchSource:0}: Error finding container bc4ab03a71538da7bb0043fc85be137dad06cb0ce19cd764cb98901227494f49: Status 404 returned error can't find the container with id bc4ab03a71538da7bb0043fc85be137dad06cb0ce19cd764cb98901227494f49 May 26 16:00:54.319470 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:54.319435 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc05844d_f6f9_4cdd_a748_a601268f1ae1.slice/crio-f5a388ca2484ce075a1e162749d62888e6a1ac467c2f5cb5e501e5783c135436 WatchSource:0}: Error finding container f5a388ca2484ce075a1e162749d62888e6a1ac467c2f5cb5e501e5783c135436: Status 404 returned error can't find the container with id f5a388ca2484ce075a1e162749d62888e6a1ac467c2f5cb5e501e5783c135436 May 26 16:00:54.320562 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:54.320538 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bb0bd9_cd9e_428c_a7dc_dc6adbb80db8.slice/crio-02bff554b2f47c4049de8df88d52666efadc2f74b0b21f417b515ddba8419903 WatchSource:0}: Error finding container 02bff554b2f47c4049de8df88d52666efadc2f74b0b21f417b515ddba8419903: Status 404 returned error can't find the container with id 02bff554b2f47c4049de8df88d52666efadc2f74b0b21f417b515ddba8419903 May 26 16:00:54.321159 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:54.321139 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fee4bfe_ae19_487c_a75b_cf4f2a013075.slice/crio-df2999121aea59730a32f06482c95e8b2b99db12774c5515e8ad4fd06c49f5ae WatchSource:0}: Error finding container df2999121aea59730a32f06482c95e8b2b99db12774c5515e8ad4fd06c49f5ae: Status 404 returned error can't find the container with id df2999121aea59730a32f06482c95e8b2b99db12774c5515e8ad4fd06c49f5ae May 26 16:00:54.322271 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:54.321997 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616802f2_6a23_4737_921e_862e05ea782c.slice/crio-59367c715378d80e923d6df1ad6bccb7bade8fb6e053aa10ac1499f874f9b8b0 WatchSource:0}: Error finding container 59367c715378d80e923d6df1ad6bccb7bade8fb6e053aa10ac1499f874f9b8b0: Status 404 returned error can't find the container with id 59367c715378d80e923d6df1ad6bccb7bade8fb6e053aa10ac1499f874f9b8b0 May 26 16:00:54.322789 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:54.322770 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5543c7e9_d0b5_444a_94bf_cf6d6bdf7f9a.slice/crio-aabf0130275b78dbc0a57f3a5973e30302068555c0e2886a94f6aa421b58b1b1 WatchSource:0}: Error finding container aabf0130275b78dbc0a57f3a5973e30302068555c0e2886a94f6aa421b58b1b1: Status 404 returned error can't find the container with id aabf0130275b78dbc0a57f3a5973e30302068555c0e2886a94f6aa421b58b1b1 May 26 16:00:54.324989 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:54.323761 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2367058a_a723_4dcc_a10d_f33b1670f57e.slice/crio-ca4d0a59a0897acba84499c637c322043273307119a4b9f0a6c0c510f8c5e18b WatchSource:0}: Error finding container ca4d0a59a0897acba84499c637c322043273307119a4b9f0a6c0c510f8c5e18b: Status 404 returned error can't find the container with id ca4d0a59a0897acba84499c637c322043273307119a4b9f0a6c0c510f8c5e18b May 26 16:00:54.326402 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:00:54.326382 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b420341_5a85_4184_9d56_616ba0fa532c.slice/crio-086152f4d473cfba1d55e6983fb967c330d4b8ce99957da8288686e232dbcfe7 WatchSource:0}: Error finding container 086152f4d473cfba1d55e6983fb967c330d4b8ce99957da8288686e232dbcfe7: Status 404 returned error can't find the container with id 086152f4d473cfba1d55e6983fb967c330d4b8ce99957da8288686e232dbcfe7 May 26 16:00:54.343903 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.343678 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:54.343994 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.343821 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:54.344049 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.344028 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs podName:264cf2a2-6d0f-4220-8797-88813b9a33fe nodeName:}" failed. No retries permitted until 2026-05-26 16:00:55.34401198 +0000 UTC m=+4.203934435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs") pod "network-metrics-daemon-vz84d" (UID: "264cf2a2-6d0f-4220-8797-88813b9a33fe") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:54.545862 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.545802 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:00:54.546037 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.545971 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:00:54.546037 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.545999 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:00:54.546037 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.546013 2573 projected.go:194] Error preparing data for projected volume kube-api-access-tk4bl for pod openshift-network-diagnostics/network-check-target-sdx75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:54.546180 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.546111 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl podName:2945e20a-dab7-473f-a55f-396c99f0d855 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:55.546091162 +0000 UTC m=+4.406013628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk4bl" (UniqueName: "kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl") pod "network-check-target-sdx75" (UID: "2945e20a-dab7-473f-a55f-396c99f0d855") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:54.646866 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.646826 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:54.646980 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.646931 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:00:54.647040 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.647003 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret podName:89ec24bc-600b-4510-a6e5-95c672ab9577 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:55.646975065 +0000 UTC m=+4.506897512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret") pod "global-pull-secret-syncer-gjfr7" (UID: "89ec24bc-600b-4510-a6e5-95c672ab9577") : object "kube-system"/"original-pull-secret" not registered May 26 16:00:54.777612 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.777565 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:52 +0000 UTC" deadline="2027-11-01 04:01:13.350657768 +0000 UTC" May 26 16:00:54.777612 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.777609 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12564h0m18.573052286s" May 26 16:00:54.887700 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.887193 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:00:54.887700 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:54.887314 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:00:54.914044 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.910709 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vn7x4" event={"ID":"7f746638-ec1d-4cc7-8e72-6f7ef6991c06","Type":"ContainerStarted","Data":"bc4ab03a71538da7bb0043fc85be137dad06cb0ce19cd764cb98901227494f49"} May 26 16:00:54.914724 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.914658 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjm54" event={"ID":"e636febb-62af-4bae-87f2-f5ba6ee79f71","Type":"ContainerStarted","Data":"be74db668bd289d570ca3d98da94e7bf76c20901759a44b298ea849b06db1e95"} May 26 16:00:54.933921 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.933353 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal" event={"ID":"5f6d5d2a73b24fd057a0b80db1b7899f","Type":"ContainerStarted","Data":"b607c35abf6be2a326a6ddbbfbfd77043c84ba5c14f07406cfc68786514d3c7e"} May 26 16:00:54.940857 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.940810 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" event={"ID":"5b420341-5a85-4184-9d56-616ba0fa532c","Type":"ContainerStarted","Data":"086152f4d473cfba1d55e6983fb967c330d4b8ce99957da8288686e232dbcfe7"} May 26 16:00:54.944089 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.944019 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7m96n" event={"ID":"616802f2-6a23-4737-921e-862e05ea782c","Type":"ContainerStarted","Data":"59367c715378d80e923d6df1ad6bccb7bade8fb6e053aa10ac1499f874f9b8b0"} May 26 16:00:54.954417 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.954306 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" event={"ID":"5fee4bfe-ae19-487c-a75b-cf4f2a013075","Type":"ContainerStarted","Data":"df2999121aea59730a32f06482c95e8b2b99db12774c5515e8ad4fd06c49f5ae"} May 26 16:00:54.964808 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.964734 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" event={"ID":"bc05844d-f6f9-4cdd-a748-a601268f1ae1","Type":"ContainerStarted","Data":"f5a388ca2484ce075a1e162749d62888e6a1ac467c2f5cb5e501e5783c135436"} May 26 16:00:54.981445 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.981412 2573 generic.go:358] "Generic (PLEG): container finished" podID="1b8861c8ce95eee7825ce03628bd1faf" containerID="1d0e8cd13efa4435903a87be7db530faf12bb3261d6ee84668362f11f1f884fa" exitCode=0 May 26 16:00:54.981679 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.981493 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" event={"ID":"1b8861c8ce95eee7825ce03628bd1faf","Type":"ContainerDied","Data":"1d0e8cd13efa4435903a87be7db530faf12bb3261d6ee84668362f11f1f884fa"} May 26 16:00:54.996783 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:54.996732 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-48.ec2.internal" podStartSLOduration=1.9967144430000001 podStartE2EDuration="1.996714443s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:54.951389878 +0000 UTC m=+3.811312350" watchObservedRunningTime="2026-05-26 16:00:54.996714443 +0000 UTC m=+3.856636914" May 26 16:00:55.007435 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:55.007335 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bqqjm" event={"ID":"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8","Type":"ContainerStarted","Data":"02bff554b2f47c4049de8df88d52666efadc2f74b0b21f417b515ddba8419903"} May 26 16:00:55.010496 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:55.010398 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" event={"ID":"2367058a-a723-4dcc-a10d-f33b1670f57e","Type":"ContainerStarted","Data":"ca4d0a59a0897acba84499c637c322043273307119a4b9f0a6c0c510f8c5e18b"} May 26 16:00:55.023039 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:55.022916 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b4gnl" event={"ID":"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a","Type":"ContainerStarted","Data":"aabf0130275b78dbc0a57f3a5973e30302068555c0e2886a94f6aa421b58b1b1"} May 26 16:00:55.353734 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:55.353672 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:55.353930 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.353872 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:55.353990 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.353939 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs podName:264cf2a2-6d0f-4220-8797-88813b9a33fe nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.35391979 +0000 UTC m=+6.213842254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs") pod "network-metrics-daemon-vz84d" (UID: "264cf2a2-6d0f-4220-8797-88813b9a33fe") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:55.555314 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:55.555274 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:00:55.555504 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.555457 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:00:55.555504 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.555482 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:00:55.555504 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.555495 2573 projected.go:194] Error preparing data for projected volume kube-api-access-tk4bl for pod openshift-network-diagnostics/network-check-target-sdx75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:55.555656 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.555560 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl podName:2945e20a-dab7-473f-a55f-396c99f0d855 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.555540666 +0000 UTC m=+6.415463130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk4bl" (UniqueName: "kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl") pod "network-check-target-sdx75" (UID: "2945e20a-dab7-473f-a55f-396c99f0d855") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:55.656300 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:55.656259 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:55.656488 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.656447 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:00:55.656547 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.656514 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret podName:89ec24bc-600b-4510-a6e5-95c672ab9577 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.65649558 +0000 UTC m=+6.516418045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret") pod "global-pull-secret-syncer-gjfr7" (UID: "89ec24bc-600b-4510-a6e5-95c672ab9577") : object "kube-system"/"original-pull-secret" not registered May 26 16:00:55.871596 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:55.870988 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:55.871596 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:55.871032 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:55.871596 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.871130 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:00:55.871596 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:55.871256 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:00:56.038133 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:56.038078 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" event={"ID":"1b8861c8ce95eee7825ce03628bd1faf","Type":"ContainerStarted","Data":"25e0f7c8ded57f9f4acc4de8b584c79bbce723a9daedb82cb3f51d45bbb8163c"} May 26 16:00:56.869336 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:56.869308 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:00:56.869552 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:56.869430 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:00:57.371952 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:57.371881 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:57.372384 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.372051 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:57.372384 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.372123 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs podName:264cf2a2-6d0f-4220-8797-88813b9a33fe nodeName:}" failed. No retries permitted until 2026-05-26 16:01:01.372103338 +0000 UTC m=+10.232025789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs") pod "network-metrics-daemon-vz84d" (UID: "264cf2a2-6d0f-4220-8797-88813b9a33fe") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:00:57.574401 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:57.573740 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:00:57.574401 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.573929 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:00:57.574401 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.573948 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:00:57.574401 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.573960 2573 projected.go:194] Error preparing data for projected volume kube-api-access-tk4bl for pod openshift-network-diagnostics/network-check-target-sdx75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:57.574401 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.574019 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl podName:2945e20a-dab7-473f-a55f-396c99f0d855 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:01.574001872 +0000 UTC m=+10.433924325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk4bl" (UniqueName: "kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl") pod "network-check-target-sdx75" (UID: "2945e20a-dab7-473f-a55f-396c99f0d855") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:00:57.674785 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:57.674247 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:57.674785 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.674409 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:00:57.674785 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.674468 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret podName:89ec24bc-600b-4510-a6e5-95c672ab9577 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:01.674448736 +0000 UTC m=+10.534371189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret") pod "global-pull-secret-syncer-gjfr7" (UID: "89ec24bc-600b-4510-a6e5-95c672ab9577") : object "kube-system"/"original-pull-secret" not registered May 26 16:00:57.870174 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:57.869496 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:57.870174 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.869628 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:00:57.870174 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:57.870032 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:57.876150 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:57.870522 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:00:58.869777 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:58.869738 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:00:58.870241 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:58.869919 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:00:59.871325 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:59.871290 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:00:59.871788 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:59.871434 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:00:59.871788 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:00:59.871462 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:00:59.871788 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:00:59.871531 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:00.869092 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:00.869053 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:00.869282 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:00.869201 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:01.412555 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:01.412516 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:01.413050 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.412668 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:01.413050 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.412740 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs podName:264cf2a2-6d0f-4220-8797-88813b9a33fe nodeName:}" failed. No retries permitted until 2026-05-26 16:01:09.412719648 +0000 UTC m=+18.272642102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs") pod "network-metrics-daemon-vz84d" (UID: "264cf2a2-6d0f-4220-8797-88813b9a33fe") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:01.614673 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:01.614628 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:01.614882 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.614804 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:01:01.614882 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.614820 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:01:01.614882 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.614846 2573 projected.go:194] Error preparing data for projected volume kube-api-access-tk4bl for pod openshift-network-diagnostics/network-check-target-sdx75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:01.615046 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.614899 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl podName:2945e20a-dab7-473f-a55f-396c99f0d855 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:09.614885082 +0000 UTC m=+18.474807529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk4bl" (UniqueName: "kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl") pod "network-check-target-sdx75" (UID: "2945e20a-dab7-473f-a55f-396c99f0d855") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:01.716016 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:01.715926 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:01.716153 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.716104 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:01:01.716205 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.716182 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret podName:89ec24bc-600b-4510-a6e5-95c672ab9577 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:09.71616667 +0000 UTC m=+18.576089141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret") pod "global-pull-secret-syncer-gjfr7" (UID: "89ec24bc-600b-4510-a6e5-95c672ab9577") : object "kube-system"/"original-pull-secret" not registered May 26 16:01:01.870985 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:01.870950 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:01.871158 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:01.871000 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:01.871158 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.871091 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:01.871556 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:01.871525 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:02.869596 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:02.869556 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:02.870116 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:02.869703 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:03.869357 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:03.869322 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:03.869357 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:03.869358 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:03.869595 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:03.869472 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:03.869652 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:03.869597 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:04.869922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:04.869886 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:04.870337 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:04.870004 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:05.869686 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:05.869650 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:05.869897 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:05.869792 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:05.869897 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:05.869865 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:05.870327 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:05.869996 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:06.869548 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:06.869510 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:06.869741 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:06.869664 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:07.869213 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:07.869178 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:07.869690 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:07.869280 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:07.869690 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:07.869348 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:07.869690 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:07.869473 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:08.869207 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:08.869174 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:08.869413 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:08.869285 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:09.476288 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:09.476257 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:09.476446 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.476373 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:09.476446 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.476430 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs podName:264cf2a2-6d0f-4220-8797-88813b9a33fe nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.476413736 +0000 UTC m=+34.336336182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs") pod "network-metrics-daemon-vz84d" (UID: "264cf2a2-6d0f-4220-8797-88813b9a33fe") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:09.677035 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:09.676999 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:09.677216 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.677152 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:01:09.677216 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.677173 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:01:09.677216 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.677185 2573 projected.go:194] Error preparing data for projected volume kube-api-access-tk4bl for pod openshift-network-diagnostics/network-check-target-sdx75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:09.677380 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.677249 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl podName:2945e20a-dab7-473f-a55f-396c99f0d855 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.677231183 +0000 UTC m=+34.537153647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk4bl" (UniqueName: "kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl") pod "network-check-target-sdx75" (UID: "2945e20a-dab7-473f-a55f-396c99f0d855") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:09.777633 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:09.777587 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:09.777808 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.777754 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:01:09.777890 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.777848 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret podName:89ec24bc-600b-4510-a6e5-95c672ab9577 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.777812682 +0000 UTC m=+34.637735157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret") pod "global-pull-secret-syncer-gjfr7" (UID: "89ec24bc-600b-4510-a6e5-95c672ab9577") : object "kube-system"/"original-pull-secret" not registered May 26 16:01:09.869194 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:09.869147 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:09.869194 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:09.869193 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:09.869410 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.869301 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:09.869461 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:09.869440 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:10.869199 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:10.869164 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:10.869369 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:10.869278 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:11.870547 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:11.870509 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:11.871035 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:11.870643 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:11.871035 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:11.870692 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:11.871035 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:11.870783 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:12.869820 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:12.869538 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:12.869938 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:12.869919 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:13.067635 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.067607 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" event={"ID":"2367058a-a723-4dcc-a10d-f33b1670f57e","Type":"ContainerStarted","Data":"9244c1f609106621b2ff7d6d4f750764f1c5cb1ce9a309aebf389ede6e850816"} May 26 16:01:13.068807 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.068785 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b4gnl" event={"ID":"5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a","Type":"ContainerStarted","Data":"ec375c43e942cfc6ebd0522baddb9058a1f7957a2c110fbc6d39dcdb54859637"} May 26 16:01:13.069852 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.069812 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vn7x4" event={"ID":"7f746638-ec1d-4cc7-8e72-6f7ef6991c06","Type":"ContainerStarted","Data":"599aa627086eea2415205005a04fb0011c5b524a3fdd653977443db521c1a5fd"} May 26 16:01:13.070959 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.070940 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjm54" event={"ID":"e636febb-62af-4bae-87f2-f5ba6ee79f71","Type":"ContainerStarted","Data":"9d4b61a2cf903e73e94facd7130c93ceacd5788713ad31480deeb4e207d2d059"} May 26 16:01:13.072871 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.072850 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" event={"ID":"5b420341-5a85-4184-9d56-616ba0fa532c","Type":"ContainerStarted","Data":"6c33043c188e6936c642f28e8d73306fe97a92dd3a146116fc03c1f28b29c507"} May 26 16:01:13.072959 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.072878 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" event={"ID":"5b420341-5a85-4184-9d56-616ba0fa532c","Type":"ContainerStarted","Data":"914e02e34803eecaec9fc349a7e5077168464ca68de73624d4f7779918e18c0d"} May 26 16:01:13.072959 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.072891 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" event={"ID":"5b420341-5a85-4184-9d56-616ba0fa532c","Type":"ContainerStarted","Data":"014e72775c048174b4d28ea061aefca06231acd13604c13819203820858db720"} May 26 16:01:13.072959 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.072903 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" event={"ID":"5b420341-5a85-4184-9d56-616ba0fa532c","Type":"ContainerStarted","Data":"08194f182fbf5f1710a359a27a0870b6e60a3c507ea28120e1ec41958290411a"} May 26 16:01:13.072959 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.072915 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" event={"ID":"5b420341-5a85-4184-9d56-616ba0fa532c","Type":"ContainerStarted","Data":"c3c586593fc797465dbb53d591407028ad15b8bf017cc29063766732749a8e45"} May 26 16:01:13.073895 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.073876 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7m96n" event={"ID":"616802f2-6a23-4737-921e-862e05ea782c","Type":"ContainerStarted","Data":"cba9b7fecd28dd7dd97a2f6bd4c451d58d39859cb58abc0064e48da442e4aa92"} May 26 16:01:13.075035 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.075010 2573 generic.go:358] "Generic (PLEG): container finished" podID="5fee4bfe-ae19-487c-a75b-cf4f2a013075" containerID="38117a01b25cd8d52f1d69c4df36919f86325edec661af78e19d17f090136cc4" exitCode=0 May 26 16:01:13.075118 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.075065 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" event={"ID":"5fee4bfe-ae19-487c-a75b-cf4f2a013075","Type":"ContainerDied","Data":"38117a01b25cd8d52f1d69c4df36919f86325edec661af78e19d17f090136cc4"} May 26 16:01:13.076496 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.076476 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" event={"ID":"bc05844d-f6f9-4cdd-a748-a601268f1ae1","Type":"ContainerStarted","Data":"3149def1dfc9fd8d3e912c82d2f2744a4fee68eaaa6a299ba0f5e6117388b2c3"} May 26 16:01:13.088385 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.088348 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-48.ec2.internal" podStartSLOduration=20.088338338 podStartE2EDuration="20.088338338s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:56.056696052 +0000 UTC m=+4.916618523" watchObservedRunningTime="2026-05-26 16:01:13.088338338 +0000 UTC m=+21.948260807" May 26 16:01:13.088629 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.088611 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8xgrm" podStartSLOduration=3.901183185 podStartE2EDuration="22.088607437s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:54.327265226 +0000 UTC m=+3.187187689" lastFinishedPulling="2026-05-26 16:01:12.514689478 +0000 UTC m=+21.374611941" observedRunningTime="2026-05-26 16:01:13.088276106 +0000 UTC m=+21.948198575" watchObservedRunningTime="2026-05-26 16:01:13.088607437 +0000 UTC m=+21.948529906" May 26 16:01:13.111447 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.111406 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-vjm54" podStartSLOduration=3.915567465 podStartE2EDuration="22.111394859s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:54.316879817 +0000 UTC m=+3.176802277" lastFinishedPulling="2026-05-26 16:01:12.512707221 +0000 UTC m=+21.372629671" observedRunningTime="2026-05-26 16:01:13.111026423 +0000 UTC m=+21.970948891" watchObservedRunningTime="2026-05-26 16:01:13.111394859 +0000 UTC m=+21.971317327" May 26 16:01:13.129265 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.129230 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7m96n" podStartSLOduration=11.899352786 podStartE2EDuration="22.12922023s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:54.324573609 +0000 UTC m=+3.184496059" lastFinishedPulling="2026-05-26 16:01:04.554441041 +0000 UTC m=+13.414363503" observedRunningTime="2026-05-26 16:01:13.129020699 +0000 UTC m=+21.988943169" watchObservedRunningTime="2026-05-26 16:01:13.12922023 +0000 UTC m=+21.989142701" May 26 16:01:13.161463 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.161419 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-vn7x4" podStartSLOduration=3.949751125 podStartE2EDuration="22.161406429s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:54.318379308 +0000 UTC m=+3.178301767" lastFinishedPulling="2026-05-26 16:01:12.530034615 +0000 UTC m=+21.389957071" observedRunningTime="2026-05-26 16:01:13.160946337 +0000 UTC m=+22.020868809" watchObservedRunningTime="2026-05-26 16:01:13.161406429 +0000 UTC m=+22.021328897" May 26 16:01:13.161631 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.161610 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b4gnl" podStartSLOduration=3.9899018059999998 podStartE2EDuration="22.161604852s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:54.327051231 +0000 UTC m=+3.186973692" lastFinishedPulling="2026-05-26 16:01:12.498754274 +0000 UTC m=+21.358676738" observedRunningTime="2026-05-26 16:01:13.144631308 +0000 UTC m=+22.004553778" watchObservedRunningTime="2026-05-26 16:01:13.161604852 +0000 UTC m=+22.021527318" May 26 16:01:13.869465 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.869436 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:13.869580 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.869464 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:13.869640 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:13.869576 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:13.870010 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:13.869984 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:13.979133 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:13.979003 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" May 26 16:01:14.080009 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.079976 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" event={"ID":"bc05844d-f6f9-4cdd-a748-a601268f1ae1","Type":"ContainerStarted","Data":"bad2a6aecf287be52cddb09ef8970ab2ad3b4372f5342aa8c7d5651960d0ef6a"} May 26 16:01:14.081513 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.081484 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bqqjm" event={"ID":"37bb0bd9-cd9e-428c-a7dc-dc6adbb80db8","Type":"ContainerStarted","Data":"508bac32b7383ce9936ada6caaf717d8bd596b66b40ce2bf2b8318f667b1087c"} May 26 16:01:14.084615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.084558 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" event={"ID":"5b420341-5a85-4184-9d56-616ba0fa532c","Type":"ContainerStarted","Data":"385fd02bc48e3158781edffcabeba062d9c30741d0a98b64304648b331634204"} May 26 16:01:14.097501 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.097449 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bqqjm" podStartSLOduration=3.922674264 podStartE2EDuration="22.09743321s" podCreationTimestamp="2026-05-26 16:00:52 +0000 UTC" firstStartedPulling="2026-05-26 16:00:54.323996645 +0000 UTC m=+3.183919094" lastFinishedPulling="2026-05-26 16:01:12.498755578 +0000 UTC m=+21.358678040" observedRunningTime="2026-05-26 16:01:14.097134427 +0000 UTC m=+22.957056897" watchObservedRunningTime="2026-05-26 16:01:14.09743321 +0000 UTC m=+22.957355686" May 26 16:01:14.568899 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.568868 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7m96n" May 26 16:01:14.569578 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.569556 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7m96n" May 26 16:01:14.805117 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.805009 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T16:01:13.979026698Z","UUID":"ab658e5b-ddb1-496e-979a-c4b9c0bccb07","Handler":null,"Name":"","Endpoint":""} May 26 16:01:14.806710 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.806671 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 May 26 16:01:14.806710 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.806709 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock May 26 16:01:14.869311 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:14.869246 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:14.869452 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:14.869347 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:15.836904 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:15.836873 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7m96n" May 26 16:01:15.837530 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:15.837494 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7m96n" May 26 16:01:15.869210 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:15.869188 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:15.869352 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:15.869310 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:15.869421 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:15.869371 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:15.869515 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:15.869475 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:16.092123 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:16.092038 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" event={"ID":"5b420341-5a85-4184-9d56-616ba0fa532c","Type":"ContainerStarted","Data":"3ae7295c600a7a8c53f6144bf2d7799267233af4e85c919f000dd6ec77231efd"} May 26 16:01:16.094220 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:16.094193 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" event={"ID":"bc05844d-f6f9-4cdd-a748-a601268f1ae1","Type":"ContainerStarted","Data":"b85a75e12cd3072110de60aee7a5151c7ce55082f514a2d9cc140087c8db2bc8"} May 26 16:01:16.113282 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:16.113227 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvp5m" podStartSLOduration=4.355640149 podStartE2EDuration="25.113209103s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:54.321312951 +0000 UTC m=+3.181235412" lastFinishedPulling="2026-05-26 16:01:15.078881916 +0000 UTC m=+23.938804366" observedRunningTime="2026-05-26 16:01:16.112562065 +0000 UTC m=+24.972484536" watchObservedRunningTime="2026-05-26 16:01:16.113209103 +0000 UTC m=+24.973131573" May 26 16:01:16.869608 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:16.869433 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:16.870049 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:16.869698 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:17.869281 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:17.869098 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:17.869423 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:17.869097 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:17.869457 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:17.869419 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:17.869510 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:17.869480 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:18.099104 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:18.099068 2573 generic.go:358] "Generic (PLEG): container finished" podID="5fee4bfe-ae19-487c-a75b-cf4f2a013075" containerID="967fc67fb0aa640ea4f628d0f1a6454c494cb8ef8d374937d2feffe9dd3ac212" exitCode=0 May 26 16:01:18.099878 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:18.099160 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" event={"ID":"5fee4bfe-ae19-487c-a75b-cf4f2a013075","Type":"ContainerDied","Data":"967fc67fb0aa640ea4f628d0f1a6454c494cb8ef8d374937d2feffe9dd3ac212"} May 26 16:01:18.102377 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:18.102353 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" event={"ID":"5b420341-5a85-4184-9d56-616ba0fa532c","Type":"ContainerStarted","Data":"94dd11ea293bb8a513892aaae550a296db30030e6ea3f625e9ff4886f8ff4b92"} May 26 16:01:18.102672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:18.102657 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:01:18.118157 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:18.118137 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:01:18.153681 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:18.153598 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" podStartSLOduration=7.792661428 podStartE2EDuration="26.153583579s" podCreationTimestamp="2026-05-26 16:00:52 +0000 UTC" firstStartedPulling="2026-05-26 16:00:54.329007455 +0000 UTC m=+3.188929915" lastFinishedPulling="2026-05-26 16:01:12.689929604 +0000 UTC m=+21.549852066" observedRunningTime="2026-05-26 16:01:18.152415628 +0000 UTC m=+27.012338097" watchObservedRunningTime="2026-05-26 16:01:18.153583579 +0000 UTC m=+27.013506047" May 26 16:01:18.869570 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:18.869545 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:18.869685 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:18.869658 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:19.106000 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.105970 2573 generic.go:358] "Generic (PLEG): container finished" podID="5fee4bfe-ae19-487c-a75b-cf4f2a013075" containerID="e7884ba0dcb7563cc6edb0b4b8e9a105bed15396537905f551bf9c5ef7f81157" exitCode=0 May 26 16:01:19.106375 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.106049 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" event={"ID":"5fee4bfe-ae19-487c-a75b-cf4f2a013075","Type":"ContainerDied","Data":"e7884ba0dcb7563cc6edb0b4b8e9a105bed15396537905f551bf9c5ef7f81157"} May 26 16:01:19.106375 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.106265 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" May 26 16:01:19.106746 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.106727 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:01:19.122244 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.122223 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:01:19.434892 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.434801 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjfr7"] May 26 16:01:19.435049 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.434927 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:19.435049 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:19.435003 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:19.437732 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.437706 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vz84d"] May 26 16:01:19.438008 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.437988 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:19.438127 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:19.438105 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:19.438389 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.438373 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdx75"] May 26 16:01:19.438466 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:19.438456 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:19.438558 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:19.438538 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:20.109604 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:20.109562 2573 generic.go:358] "Generic (PLEG): container finished" podID="5fee4bfe-ae19-487c-a75b-cf4f2a013075" containerID="0be78cd2447c0cbf1288596ef7387a1d0e6bc4443465e11a85bc5e2ae083c935" exitCode=0 May 26 16:01:20.109987 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:20.109643 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" event={"ID":"5fee4bfe-ae19-487c-a75b-cf4f2a013075","Type":"ContainerDied","Data":"0be78cd2447c0cbf1288596ef7387a1d0e6bc4443465e11a85bc5e2ae083c935"} May 26 16:01:20.109987 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:20.109812 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" May 26 16:01:20.869523 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:20.869491 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:20.869698 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:20.869607 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:21.111802 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:21.111766 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" May 26 16:01:21.870290 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:21.870198 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:21.870447 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:21.870320 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:21.870561 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:21.870534 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:21.870682 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:21.870653 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:22.869708 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:22.869668 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:22.870163 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:22.869796 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:23.548979 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:23.548947 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:01:23.549222 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:23.549204 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" May 26 16:01:23.561054 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:23.560892 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" podUID="5b420341-5a85-4184-9d56-616ba0fa532c" containerName="ovnkube-controller" probeResult="failure" output="" May 26 16:01:23.569629 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:23.569589 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" podUID="5b420341-5a85-4184-9d56-616ba0fa532c" containerName="ovnkube-controller" probeResult="failure" output="" May 26 16:01:23.869572 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:23.869490 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:23.869723 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:23.869634 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjfr7" podUID="89ec24bc-600b-4510-a6e5-95c672ab9577" May 26 16:01:23.870073 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:23.870040 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:23.870376 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:23.870182 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vz84d" podUID="264cf2a2-6d0f-4220-8797-88813b9a33fe" May 26 16:01:24.869471 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:24.869434 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:24.869728 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:24.869553 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx75" podUID="2945e20a-dab7-473f-a55f-396c99f0d855" May 26 16:01:25.425494 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.425465 2573 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-48.ec2.internal" event="NodeReady" May 26 16:01:25.426005 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.425595 2573 kubelet_node_status.go:550] "Fast updating node status as it just became ready" May 26 16:01:25.478966 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.478932 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p8lq6"] May 26 16:01:25.501143 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.501114 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:25.501328 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.501270 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:25.501397 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.501345 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs podName:264cf2a2-6d0f-4220-8797-88813b9a33fe nodeName:}" failed. No retries permitted until 2026-05-26 16:01:57.501325834 +0000 UTC m=+66.361248282 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs") pod "network-metrics-daemon-vz84d" (UID: "264cf2a2-6d0f-4220-8797-88813b9a33fe") : object "openshift-multus"/"metrics-daemon-secret" not registered May 26 16:01:25.529365 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.529329 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rs5nq"] May 26 16:01:25.529527 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.529498 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.532117 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.532081 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" May 26 16:01:25.532265 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.532092 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" May 26 16:01:25.532265 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.532138 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w25cq\"" May 26 16:01:25.544327 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.544308 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p8lq6"] May 26 16:01:25.544410 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.544334 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rs5nq"] May 26 16:01:25.544447 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.544413 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:25.548510 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.548490 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" May 26 16:01:25.548621 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.548573 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" May 26 16:01:25.548621 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.548595 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" May 26 16:01:25.548723 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.548678 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l7p5g\"" May 26 16:01:25.703257 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.703169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42e8c71-312c-41a0-a8d9-b116879302ae-config-volume\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.703257 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.703216 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42e8c71-312c-41a0-a8d9-b116879302ae-tmp-dir\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.703497 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.703312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.703497 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.703344 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:25.703497 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.703373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwf7\" (UniqueName: \"kubernetes.io/projected/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-kube-api-access-kqwf7\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:25.703497 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.703486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-249t7\" (UniqueName: \"kubernetes.io/projected/c42e8c71-312c-41a0-a8d9-b116879302ae-kube-api-access-249t7\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.703662 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.703552 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:25.703698 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.703665 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 26 16:01:25.703698 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.703680 2573 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 26 16:01:25.703698 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.703689 2573 projected.go:194] Error preparing data for projected volume kube-api-access-tk4bl for pod openshift-network-diagnostics/network-check-target-sdx75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:25.703784 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.703734 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl podName:2945e20a-dab7-473f-a55f-396c99f0d855 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:57.703721214 +0000 UTC m=+66.563643661 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk4bl" (UniqueName: "kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl") pod "network-check-target-sdx75" (UID: "2945e20a-dab7-473f-a55f-396c99f0d855") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 26 16:01:25.804198 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.804156 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.804198 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.804195 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.804212 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwf7\" (UniqueName: \"kubernetes.io/projected/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-kube-api-access-kqwf7\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.804235 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-249t7\" (UniqueName: \"kubernetes.io/projected/c42e8c71-312c-41a0-a8d9-b116879302ae-kube-api-access-249t7\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.804285 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.804307 2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.804315 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42e8c71-312c-41a0-a8d9-b116879302ae-config-volume\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.804307 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.804374 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert podName:ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:26.304353289 +0000 UTC m=+35.164275739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert") pod "ingress-canary-rs5nq" (UID: "ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a") : secret "canary-serving-cert" not found May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.804387 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.804413 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls podName:c42e8c71-312c-41a0-a8d9-b116879302ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:26.304394095 +0000 UTC m=+35.164316553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls") pod "dns-default-p8lq6" (UID: "c42e8c71-312c-41a0-a8d9-b116879302ae") : secret "dns-default-metrics-tls" not found May 26 16:01:25.804431 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:25.804435 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret podName:89ec24bc-600b-4510-a6e5-95c672ab9577 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:57.804419592 +0000 UTC m=+66.664342049 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret") pod "global-pull-secret-syncer-gjfr7" (UID: "89ec24bc-600b-4510-a6e5-95c672ab9577") : object "kube-system"/"original-pull-secret" not registered May 26 16:01:25.804918 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.804457 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42e8c71-312c-41a0-a8d9-b116879302ae-tmp-dir\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.804918 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.804754 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42e8c71-312c-41a0-a8d9-b116879302ae-config-volume\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.804918 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.804793 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42e8c71-312c-41a0-a8d9-b116879302ae-tmp-dir\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.814955 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.814932 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-249t7\" (UniqueName: \"kubernetes.io/projected/c42e8c71-312c-41a0-a8d9-b116879302ae-kube-api-access-249t7\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:25.815240 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.815219 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwf7\" (UniqueName: \"kubernetes.io/projected/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-kube-api-access-kqwf7\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:25.869249 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.869209 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:25.869426 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.869227 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:25.871970 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.871949 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 26 16:01:25.872106 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.871970 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsz4f\"" May 26 16:01:25.872106 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:25.871970 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 26 16:01:26.307291 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:26.307266 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:26.307432 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:26.307307 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:26.307432 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:26.307401 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:26.307432 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:26.307417 2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:26.307608 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:26.307474 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls podName:c42e8c71-312c-41a0-a8d9-b116879302ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.307457502 +0000 UTC m=+36.167379949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls") pod "dns-default-p8lq6" (UID: "c42e8c71-312c-41a0-a8d9-b116879302ae") : secret "dns-default-metrics-tls" not found May 26 16:01:26.307608 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:26.307500 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert podName:ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.307487155 +0000 UTC m=+36.167409605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert") pod "ingress-canary-rs5nq" (UID: "ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a") : secret "canary-serving-cert" not found May 26 16:01:26.868967 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:26.868927 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:26.872085 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:26.872062 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 26 16:01:26.872191 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:26.872150 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 26 16:01:26.872260 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:26.872195 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf4w7\"" May 26 16:01:27.124453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:27.124358 2573 generic.go:358] "Generic (PLEG): container finished" podID="5fee4bfe-ae19-487c-a75b-cf4f2a013075" containerID="537e2c014d81c455e41c2bf00060ea06be5a5e19ab04c2bf8b5c10a42b99990e" exitCode=0 May 26 16:01:27.124453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:27.124394 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" event={"ID":"5fee4bfe-ae19-487c-a75b-cf4f2a013075","Type":"ContainerDied","Data":"537e2c014d81c455e41c2bf00060ea06be5a5e19ab04c2bf8b5c10a42b99990e"} May 26 16:01:27.313800 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:27.313775 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:27.313919 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:27.313871 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:27.313977 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:27.313950 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:27.313977 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:27.313952 2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:27.314063 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:27.314012 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls podName:c42e8c71-312c-41a0-a8d9-b116879302ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.313998859 +0000 UTC m=+38.173921306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls") pod "dns-default-p8lq6" (UID: "c42e8c71-312c-41a0-a8d9-b116879302ae") : secret "dns-default-metrics-tls" not found May 26 16:01:27.314063 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:27.314025 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert podName:ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.314019651 +0000 UTC m=+38.173942098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert") pod "ingress-canary-rs5nq" (UID: "ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a") : secret "canary-serving-cert" not found May 26 16:01:28.129232 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:28.129199 2573 generic.go:358] "Generic (PLEG): container finished" podID="5fee4bfe-ae19-487c-a75b-cf4f2a013075" containerID="29b15cd5b6ed83cd4d2f13461b4170ef6bbdb29c7e47109622cee494dcd891cb" exitCode=0 May 26 16:01:28.129669 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:28.129242 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" event={"ID":"5fee4bfe-ae19-487c-a75b-cf4f2a013075","Type":"ContainerDied","Data":"29b15cd5b6ed83cd4d2f13461b4170ef6bbdb29c7e47109622cee494dcd891cb"} May 26 16:01:29.133977 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:29.133948 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" event={"ID":"5fee4bfe-ae19-487c-a75b-cf4f2a013075","Type":"ContainerStarted","Data":"e9870d9fe9e0d486b2cbdd4fe1fd3b08bea8fb826b3c40e9ea20645371612593"} May 26 16:01:29.159986 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:29.159939 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r4lmb" podStartSLOduration=6.302817348 podStartE2EDuration="38.159925428s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:00:54.32436482 +0000 UTC m=+3.184287281" lastFinishedPulling="2026-05-26 16:01:26.181472911 +0000 UTC m=+35.041395361" observedRunningTime="2026-05-26 16:01:29.159452934 +0000 UTC m=+38.019375406" watchObservedRunningTime="2026-05-26 16:01:29.159925428 +0000 UTC m=+38.019847896" May 26 16:01:29.327614 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:29.327578 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:29.327614 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:29.327616 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:29.327889 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:29.327708 2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:29.327889 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:29.327731 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:29.327889 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:29.327758 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert podName:ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:33.327745198 +0000 UTC m=+42.187667646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert") pod "ingress-canary-rs5nq" (UID: "ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a") : secret "canary-serving-cert" not found May 26 16:01:29.327889 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:29.327791 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls podName:c42e8c71-312c-41a0-a8d9-b116879302ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:33.327774717 +0000 UTC m=+42.187697171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls") pod "dns-default-p8lq6" (UID: "c42e8c71-312c-41a0-a8d9-b116879302ae") : secret "dns-default-metrics-tls" not found May 26 16:01:33.360654 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:33.360430 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:33.361064 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:33.360677 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:33.361064 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:33.360589 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:33.361064 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:33.360783 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls podName:c42e8c71-312c-41a0-a8d9-b116879302ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:41.360762557 +0000 UTC m=+50.220685023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls") pod "dns-default-p8lq6" (UID: "c42e8c71-312c-41a0-a8d9-b116879302ae") : secret "dns-default-metrics-tls" not found May 26 16:01:33.361064 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:33.360822 2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:33.361064 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:33.360886 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert podName:ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:41.360873635 +0000 UTC m=+50.220796097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert") pod "ingress-canary-rs5nq" (UID: "ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a") : secret "canary-serving-cert" not found May 26 16:01:41.416661 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:41.416615 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:41.417145 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:41.416669 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:41.417145 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:41.416790 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:41.417145 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:41.416876 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls podName:c42e8c71-312c-41a0-a8d9-b116879302ae nodeName:}" failed. No retries permitted until 2026-05-26 16:01:57.416858258 +0000 UTC m=+66.276780709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls") pod "dns-default-p8lq6" (UID: "c42e8c71-312c-41a0-a8d9-b116879302ae") : secret "dns-default-metrics-tls" not found May 26 16:01:41.417145 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:41.416790 2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:41.417145 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:41.416936 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert podName:ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:57.416925509 +0000 UTC m=+66.276847970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert") pod "ingress-canary-rs5nq" (UID: "ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a") : secret "canary-serving-cert" not found May 26 16:01:53.570488 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:53.570453 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9pgwm" May 26 16:01:57.427049 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.427013 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:01:57.427049 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.427053 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:01:57.427465 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:57.427154 2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:01:57.427465 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:57.427204 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert podName:ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:29.427189319 +0000 UTC m=+98.287111766 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert") pod "ingress-canary-rs5nq" (UID: "ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a") : secret "canary-serving-cert" not found May 26 16:01:57.427465 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:57.427248 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:01:57.427465 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:57.427297 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls podName:c42e8c71-312c-41a0-a8d9-b116879302ae nodeName:}" failed. No retries permitted until 2026-05-26 16:02:29.427283791 +0000 UTC m=+98.287206263 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls") pod "dns-default-p8lq6" (UID: "c42e8c71-312c-41a0-a8d9-b116879302ae") : secret "dns-default-metrics-tls" not found May 26 16:01:57.528120 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.528081 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:01:57.530411 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.530391 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 26 16:01:57.539236 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:57.539217 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found May 26 16:01:57.539298 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:01:57.539271 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs podName:264cf2a2-6d0f-4220-8797-88813b9a33fe nodeName:}" failed. No retries permitted until 2026-05-26 16:03:01.53925582 +0000 UTC m=+130.399178268 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs") pod "network-metrics-daemon-vz84d" (UID: "264cf2a2-6d0f-4220-8797-88813b9a33fe") : secret "metrics-daemon-secret" not found May 26 16:01:57.729741 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.729651 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:57.733669 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.733650 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 26 16:01:57.743114 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.743094 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 26 16:01:57.754151 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.754124 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4bl\" (UniqueName: \"kubernetes.io/projected/2945e20a-dab7-473f-a55f-396c99f0d855-kube-api-access-tk4bl\") pod \"network-check-target-sdx75\" (UID: \"2945e20a-dab7-473f-a55f-396c99f0d855\") " pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:57.779951 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.779922 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf4w7\"" May 26 16:01:57.788112 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.788093 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:01:57.830274 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.830240 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:57.833339 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.833318 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 26 16:01:57.843417 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.843395 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89ec24bc-600b-4510-a6e5-95c672ab9577-original-pull-secret\") pod \"global-pull-secret-syncer-gjfr7\" (UID: \"89ec24bc-600b-4510-a6e5-95c672ab9577\") " pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:57.978573 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.978540 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjfr7" May 26 16:01:57.985517 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:57.985463 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdx75"] May 26 16:01:57.990242 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:01:57.990216 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2945e20a_dab7_473f_a55f_396c99f0d855.slice/crio-b4cba716d1853721c26ca923fc0fc840fbb861ab65b6dbcd1ffe7f1c6a52517c WatchSource:0}: Error finding container b4cba716d1853721c26ca923fc0fc840fbb861ab65b6dbcd1ffe7f1c6a52517c: Status 404 returned error can't find the container with id b4cba716d1853721c26ca923fc0fc840fbb861ab65b6dbcd1ffe7f1c6a52517c May 26 16:01:58.097801 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:58.097767 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjfr7"] May 26 16:01:58.101658 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:01:58.101633 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ec24bc_600b_4510_a6e5_95c672ab9577.slice/crio-9118a24d3e6d6b3ecca18cd6ba5399c56dc8d463c03ad040421d82c5778fd2fa WatchSource:0}: Error finding container 9118a24d3e6d6b3ecca18cd6ba5399c56dc8d463c03ad040421d82c5778fd2fa: Status 404 returned error can't find the container with id 9118a24d3e6d6b3ecca18cd6ba5399c56dc8d463c03ad040421d82c5778fd2fa May 26 16:01:58.187934 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:58.187894 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdx75" event={"ID":"2945e20a-dab7-473f-a55f-396c99f0d855","Type":"ContainerStarted","Data":"b4cba716d1853721c26ca923fc0fc840fbb861ab65b6dbcd1ffe7f1c6a52517c"} May 26 16:01:58.188814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:01:58.188789 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjfr7" event={"ID":"89ec24bc-600b-4510-a6e5-95c672ab9577","Type":"ContainerStarted","Data":"9118a24d3e6d6b3ecca18cd6ba5399c56dc8d463c03ad040421d82c5778fd2fa"} May 26 16:02:03.199597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:03.199553 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjfr7" event={"ID":"89ec24bc-600b-4510-a6e5-95c672ab9577","Type":"ContainerStarted","Data":"78d2f8cb1f4a3753fb911c4ea2d002d4f58b72b3a42d890affbec180a8ab8cd9"} May 26 16:02:03.200715 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:03.200687 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdx75" event={"ID":"2945e20a-dab7-473f-a55f-396c99f0d855","Type":"ContainerStarted","Data":"18348abdfb62af27111d5c4b428f363ff5d73a7647f0c4f7d502b5a85f83167e"} May 26 16:02:03.200845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:03.200817 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:02:03.216695 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:03.216656 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gjfr7" podStartSLOduration=66.103181729 podStartE2EDuration="1m10.216644193s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:01:58.103316966 +0000 UTC m=+66.963239418" lastFinishedPulling="2026-05-26 16:02:02.216779429 +0000 UTC m=+71.076701882" observedRunningTime="2026-05-26 16:02:03.216104592 +0000 UTC m=+72.076027064" watchObservedRunningTime="2026-05-26 16:02:03.216644193 +0000 UTC m=+72.076566662" May 26 16:02:03.231920 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:03.231883 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sdx75" podStartSLOduration=67.010067923 podStartE2EDuration="1m11.231873969s" podCreationTimestamp="2026-05-26 16:00:52 +0000 UTC" firstStartedPulling="2026-05-26 16:01:57.992551706 +0000 UTC m=+66.852474153" lastFinishedPulling="2026-05-26 16:02:02.214357746 +0000 UTC m=+71.074280199" observedRunningTime="2026-05-26 16:02:03.231295362 +0000 UTC m=+72.091217832" watchObservedRunningTime="2026-05-26 16:02:03.231873969 +0000 UTC m=+72.091796437" May 26 16:02:29.448789 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:29.448751 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:02:29.449235 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:29.448823 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:02:29.449235 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:29.448928 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 26 16:02:29.449235 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:29.448931 2573 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 26 16:02:29.449235 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:29.448995 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls podName:c42e8c71-312c-41a0-a8d9-b116879302ae nodeName:}" failed. No retries permitted until 2026-05-26 16:03:33.448978189 +0000 UTC m=+162.308900636 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls") pod "dns-default-p8lq6" (UID: "c42e8c71-312c-41a0-a8d9-b116879302ae") : secret "dns-default-metrics-tls" not found May 26 16:02:29.449235 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:29.449009 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert podName:ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a nodeName:}" failed. No retries permitted until 2026-05-26 16:03:33.449003208 +0000 UTC m=+162.308925655 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert") pod "ingress-canary-rs5nq" (UID: "ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a") : secret "canary-serving-cert" not found May 26 16:02:34.205055 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:34.205021 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sdx75" May 26 16:02:42.709643 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.709610 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9"] May 26 16:02:42.711213 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.711197 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.715429 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.714500 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" May 26 16:02:42.715429 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.714512 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" May 26 16:02:42.715429 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.715168 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d6lz5\"" May 26 16:02:42.715429 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.715389 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" May 26 16:02:42.715429 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.715423 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-psh92"] May 26 16:02:42.715790 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.715484 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" May 26 16:02:42.717657 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.717608 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-tbzzs"] May 26 16:02:42.718078 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.718058 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.719439 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.719421 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-95767f565-hd7l2"] May 26 16:02:42.719606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.719588 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.720454 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.720429 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" May 26 16:02:42.720607 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.720480 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" May 26 16:02:42.721205 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.720714 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" May 26 16:02:42.721508 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.721486 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" May 26 16:02:42.721611 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.721490 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-slh98\"" May 26 16:02:42.721611 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.721573 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.723637 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.723618 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9"] May 26 16:02:42.728122 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.728101 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" May 26 16:02:42.728223 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.728158 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" May 26 16:02:42.728276 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.728249 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" May 26 16:02:42.732982 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.732964 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9w8qp\"" May 26 16:02:42.733080 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.732980 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6l5h\"" May 26 16:02:42.733606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.733556 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" May 26 16:02:42.733801 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.733775 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" May 26 16:02:42.733993 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.733821 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" May 26 16:02:42.734179 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.734131 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" May 26 16:02:42.734382 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.734346 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-psh92"] May 26 16:02:42.735568 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.735403 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-tbzzs"] May 26 16:02:42.735646 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.735624 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" May 26 16:02:42.736541 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.736524 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" May 26 16:02:42.737023 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.737008 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" May 26 16:02:42.740265 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.740242 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-95767f565-hd7l2"] May 26 16:02:42.796283 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.796251 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz"] May 26 16:02:42.797960 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.797944 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz" May 26 16:02:42.800123 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.800104 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" May 26 16:02:42.800230 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.800163 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fr2lh\"" May 26 16:02:42.800357 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.800340 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" May 26 16:02:42.811171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.811151 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz"] May 26 16:02:42.835432 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a782d182-009d-4d16-babd-26586d23986b-serving-cert\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.835538 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.835538 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kzn6\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-kube-api-access-5kzn6\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.835538 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/317da9eb-b20e-413a-af49-fde829ee35d5-tmp\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.835538 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/317da9eb-b20e-413a-af49-fde829ee35d5-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.835672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835542 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-bound-sa-token\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.835672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/317da9eb-b20e-413a-af49-fde829ee35d5-serving-cert\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.835672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835616 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/317da9eb-b20e-413a-af49-fde829ee35d5-service-ca-bundle\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.835672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835651 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a782d182-009d-4d16-babd-26586d23986b-trusted-ca\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.835672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835669 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-trusted-ca\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.835814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26qn\" (UniqueName: \"kubernetes.io/projected/317da9eb-b20e-413a-af49-fde829ee35d5-kube-api-access-t26qn\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.835814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-certificates\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.835814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ade83d5-dc58-49a9-b12d-0ef025b0d373-config\") pod \"kube-storage-version-migrator-operator-565644f49-x8qn9\" (UID: \"5ade83d5-dc58-49a9-b12d-0ef025b0d373\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.835814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/317da9eb-b20e-413a-af49-fde829ee35d5-snapshots\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.835814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a782d182-009d-4d16-babd-26586d23986b-config\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.835814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835799 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-image-registry-private-configuration\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.836004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ade83d5-dc58-49a9-b12d-0ef025b0d373-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-x8qn9\" (UID: \"5ade83d5-dc58-49a9-b12d-0ef025b0d373\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.836004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-installation-pull-secrets\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.836004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkp5d\" (UniqueName: \"kubernetes.io/projected/a782d182-009d-4d16-babd-26586d23986b-kube-api-access-rkp5d\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.836004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqjb\" (UniqueName: \"kubernetes.io/projected/5ade83d5-dc58-49a9-b12d-0ef025b0d373-kube-api-access-hwqjb\") pod \"kube-storage-version-migrator-operator-565644f49-x8qn9\" (UID: \"5ade83d5-dc58-49a9-b12d-0ef025b0d373\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.836004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.835944 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-ca-trust-extracted\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.936234 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936206 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-trusted-ca\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.936359 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936247 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t26qn\" (UniqueName: \"kubernetes.io/projected/317da9eb-b20e-413a-af49-fde829ee35d5-kube-api-access-t26qn\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.936359 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936275 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-certificates\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.936359 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936300 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ade83d5-dc58-49a9-b12d-0ef025b0d373-config\") pod \"kube-storage-version-migrator-operator-565644f49-x8qn9\" (UID: \"5ade83d5-dc58-49a9-b12d-0ef025b0d373\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.936359 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chgw7\" (UniqueName: \"kubernetes.io/projected/27f5c521-61a5-4184-855b-243fd8577783-kube-api-access-chgw7\") pod \"volume-data-source-validator-5cf8c89d64-flgsz\" (UID: \"27f5c521-61a5-4184-855b-243fd8577783\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz" May 26 16:02:42.936558 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936368 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/317da9eb-b20e-413a-af49-fde829ee35d5-snapshots\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.936558 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936393 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a782d182-009d-4d16-babd-26586d23986b-config\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.936558 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936419 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-image-registry-private-configuration\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.936558 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936449 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ade83d5-dc58-49a9-b12d-0ef025b0d373-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-x8qn9\" (UID: \"5ade83d5-dc58-49a9-b12d-0ef025b0d373\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.936558 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936475 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-installation-pull-secrets\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.936558 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936502 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rkp5d\" (UniqueName: \"kubernetes.io/projected/a782d182-009d-4d16-babd-26586d23986b-kube-api-access-rkp5d\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.936558 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936548 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqjb\" (UniqueName: \"kubernetes.io/projected/5ade83d5-dc58-49a9-b12d-0ef025b0d373-kube-api-access-hwqjb\") pod \"kube-storage-version-migrator-operator-565644f49-x8qn9\" (UID: \"5ade83d5-dc58-49a9-b12d-0ef025b0d373\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936575 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-ca-trust-extracted\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936624 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a782d182-009d-4d16-babd-26586d23986b-serving-cert\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936648 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936678 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5kzn6\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-kube-api-access-5kzn6\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936714 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/317da9eb-b20e-413a-af49-fde829ee35d5-tmp\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936736 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/317da9eb-b20e-413a-af49-fde829ee35d5-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936760 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-bound-sa-token\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936791 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/317da9eb-b20e-413a-af49-fde829ee35d5-serving-cert\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936820 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/317da9eb-b20e-413a-af49-fde829ee35d5-service-ca-bundle\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.936922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.936894 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a782d182-009d-4d16-babd-26586d23986b-trusted-ca\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.937376 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.937008 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-certificates\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.937376 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.937036 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ade83d5-dc58-49a9-b12d-0ef025b0d373-config\") pod \"kube-storage-version-migrator-operator-565644f49-x8qn9\" (UID: \"5ade83d5-dc58-49a9-b12d-0ef025b0d373\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.937376 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.937120 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a782d182-009d-4d16-babd-26586d23986b-config\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.937376 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:42.937133 2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:02:42.937376 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:42.937149 2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95767f565-hd7l2: secret "image-registry-tls" not found May 26 16:02:42.937376 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:42.937201 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls podName:8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:43.437183213 +0000 UTC m=+112.297105663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls") pod "image-registry-95767f565-hd7l2" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1") : secret "image-registry-tls" not found May 26 16:02:42.937376 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.937209 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/317da9eb-b20e-413a-af49-fde829ee35d5-snapshots\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.937727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.937644 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a782d182-009d-4d16-babd-26586d23986b-trusted-ca\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.937727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.937702 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-ca-trust-extracted\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.938390 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.937964 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-trusted-ca\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.938390 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.938156 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/317da9eb-b20e-413a-af49-fde829ee35d5-service-ca-bundle\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.938390 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.938234 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/317da9eb-b20e-413a-af49-fde829ee35d5-tmp\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.939011 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.938989 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/317da9eb-b20e-413a-af49-fde829ee35d5-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.939215 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.939197 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ade83d5-dc58-49a9-b12d-0ef025b0d373-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-x8qn9\" (UID: \"5ade83d5-dc58-49a9-b12d-0ef025b0d373\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.939933 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.939906 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/317da9eb-b20e-413a-af49-fde829ee35d5-serving-cert\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.939933 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.939923 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-image-registry-private-configuration\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.940072 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.940030 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-installation-pull-secrets\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.940342 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.940325 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a782d182-009d-4d16-babd-26586d23986b-serving-cert\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.949273 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.949245 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkp5d\" (UniqueName: \"kubernetes.io/projected/a782d182-009d-4d16-babd-26586d23986b-kube-api-access-rkp5d\") pod \"console-operator-76dff48d59-psh92\" (UID: \"a782d182-009d-4d16-babd-26586d23986b\") " pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:42.949273 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.949264 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqjb\" (UniqueName: \"kubernetes.io/projected/5ade83d5-dc58-49a9-b12d-0ef025b0d373-kube-api-access-hwqjb\") pod \"kube-storage-version-migrator-operator-565644f49-x8qn9\" (UID: \"5ade83d5-dc58-49a9-b12d-0ef025b0d373\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:42.949423 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.949405 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26qn\" (UniqueName: \"kubernetes.io/projected/317da9eb-b20e-413a-af49-fde829ee35d5-kube-api-access-t26qn\") pod \"insights-operator-78bdfb888c-tbzzs\" (UID: \"317da9eb-b20e-413a-af49-fde829ee35d5\") " pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:42.949988 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.949970 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-bound-sa-token\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:42.951681 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:42.951660 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kzn6\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-kube-api-access-5kzn6\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:43.024453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.024430 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" May 26 16:02:43.031136 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.031116 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:43.037893 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.037873 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-chgw7\" (UniqueName: \"kubernetes.io/projected/27f5c521-61a5-4184-855b-243fd8577783-kube-api-access-chgw7\") pod \"volume-data-source-validator-5cf8c89d64-flgsz\" (UID: \"27f5c521-61a5-4184-855b-243fd8577783\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz" May 26 16:02:43.038220 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.038195 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" May 26 16:02:43.048348 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.048309 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgw7\" (UniqueName: \"kubernetes.io/projected/27f5c521-61a5-4184-855b-243fd8577783-kube-api-access-chgw7\") pod \"volume-data-source-validator-5cf8c89d64-flgsz\" (UID: \"27f5c521-61a5-4184-855b-243fd8577783\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz" May 26 16:02:43.105943 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.105912 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz" May 26 16:02:43.183772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.183704 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9"] May 26 16:02:43.187307 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:02:43.187285 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ade83d5_dc58_49a9_b12d_0ef025b0d373.slice/crio-758eafecb13243f6e15e1e0b246bb27c0c25f7c12ff1ee5f1239be3d9ac6ac02 WatchSource:0}: Error finding container 758eafecb13243f6e15e1e0b246bb27c0c25f7c12ff1ee5f1239be3d9ac6ac02: Status 404 returned error can't find the container with id 758eafecb13243f6e15e1e0b246bb27c0c25f7c12ff1ee5f1239be3d9ac6ac02 May 26 16:02:43.237805 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.237778 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz"] May 26 16:02:43.241177 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:02:43.241154 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f5c521_61a5_4184_855b_243fd8577783.slice/crio-71f526c1dba8eed83614ab852e7228d115225f0e754a6653adf271cd49a1af84 WatchSource:0}: Error finding container 71f526c1dba8eed83614ab852e7228d115225f0e754a6653adf271cd49a1af84: Status 404 returned error can't find the container with id 71f526c1dba8eed83614ab852e7228d115225f0e754a6653adf271cd49a1af84 May 26 16:02:43.278258 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.278195 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" event={"ID":"5ade83d5-dc58-49a9-b12d-0ef025b0d373","Type":"ContainerStarted","Data":"758eafecb13243f6e15e1e0b246bb27c0c25f7c12ff1ee5f1239be3d9ac6ac02"} May 26 16:02:43.279009 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.278989 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz" event={"ID":"27f5c521-61a5-4184-855b-243fd8577783","Type":"ContainerStarted","Data":"71f526c1dba8eed83614ab852e7228d115225f0e754a6653adf271cd49a1af84"} May 26 16:02:43.389952 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.389922 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-tbzzs"] May 26 16:02:43.390632 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.390614 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-psh92"] May 26 16:02:43.394460 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:02:43.394433 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda782d182_009d_4d16_babd_26586d23986b.slice/crio-c9bc9c310d2c970b2f5a58d4eb5ce7883d85248f50ddd6308895ecdf3642f751 WatchSource:0}: Error finding container c9bc9c310d2c970b2f5a58d4eb5ce7883d85248f50ddd6308895ecdf3642f751: Status 404 returned error can't find the container with id c9bc9c310d2c970b2f5a58d4eb5ce7883d85248f50ddd6308895ecdf3642f751 May 26 16:02:43.394898 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:02:43.394875 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317da9eb_b20e_413a_af49_fde829ee35d5.slice/crio-399065a248bb92d782b05bc7b349155fb95062f2dc1388274bf76519ea82f6c9 WatchSource:0}: Error finding container 399065a248bb92d782b05bc7b349155fb95062f2dc1388274bf76519ea82f6c9: Status 404 returned error can't find the container with id 399065a248bb92d782b05bc7b349155fb95062f2dc1388274bf76519ea82f6c9 May 26 16:02:43.440128 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:43.440103 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:43.440220 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:43.440209 2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:02:43.440263 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:43.440223 2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95767f565-hd7l2: secret "image-registry-tls" not found May 26 16:02:43.440263 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:43.440261 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls podName:8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:44.440248675 +0000 UTC m=+113.300171123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls") pod "image-registry-95767f565-hd7l2" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1") : secret "image-registry-tls" not found May 26 16:02:44.282533 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:44.282480 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-psh92" event={"ID":"a782d182-009d-4d16-babd-26586d23986b","Type":"ContainerStarted","Data":"c9bc9c310d2c970b2f5a58d4eb5ce7883d85248f50ddd6308895ecdf3642f751"} May 26 16:02:44.284532 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:44.284494 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" event={"ID":"317da9eb-b20e-413a-af49-fde829ee35d5","Type":"ContainerStarted","Data":"399065a248bb92d782b05bc7b349155fb95062f2dc1388274bf76519ea82f6c9"} May 26 16:02:44.446727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:44.446541 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:44.447002 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:44.446763 2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:02:44.447002 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:44.446780 2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95767f565-hd7l2: secret "image-registry-tls" not found May 26 16:02:44.447002 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:44.446875 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls podName:8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:46.446857884 +0000 UTC m=+115.306780332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls") pod "image-registry-95767f565-hd7l2" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1") : secret "image-registry-tls" not found May 26 16:02:45.794114 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:45.794077 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s"] May 26 16:02:45.796899 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:45.796880 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s" May 26 16:02:45.799261 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:45.799238 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7pjvs\"" May 26 16:02:45.807615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:45.807594 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s"] May 26 16:02:45.861052 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:45.861011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftkq5\" (UniqueName: \"kubernetes.io/projected/082eaf67-c961-4b94-8704-846ebd2e616e-kube-api-access-ftkq5\") pod \"network-check-source-7f44b6dd87-f4x4s\" (UID: \"082eaf67-c961-4b94-8704-846ebd2e616e\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s" May 26 16:02:45.961927 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:45.961884 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ftkq5\" (UniqueName: \"kubernetes.io/projected/082eaf67-c961-4b94-8704-846ebd2e616e-kube-api-access-ftkq5\") pod \"network-check-source-7f44b6dd87-f4x4s\" (UID: \"082eaf67-c961-4b94-8704-846ebd2e616e\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s" May 26 16:02:45.971318 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:45.971287 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftkq5\" (UniqueName: \"kubernetes.io/projected/082eaf67-c961-4b94-8704-846ebd2e616e-kube-api-access-ftkq5\") pod \"network-check-source-7f44b6dd87-f4x4s\" (UID: \"082eaf67-c961-4b94-8704-846ebd2e616e\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s" May 26 16:02:46.106799 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:46.106713 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s" May 26 16:02:46.466195 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:46.466159 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:46.466353 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:46.466328 2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:02:46.466353 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:46.466352 2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95767f565-hd7l2: secret "image-registry-tls" not found May 26 16:02:46.466515 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:46.466417 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls podName:8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:50.466395991 +0000 UTC m=+119.326318452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls") pod "image-registry-95767f565-hd7l2" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1") : secret "image-registry-tls" not found May 26 16:02:46.573908 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:46.573736 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s"] May 26 16:02:46.578869 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:02:46.578786 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082eaf67_c961_4b94_8704_846ebd2e616e.slice/crio-57346300794b23744d58bfc9bc9b4d15329fdb33f231b0441ebde76df3c7eddb WatchSource:0}: Error finding container 57346300794b23744d58bfc9bc9b4d15329fdb33f231b0441ebde76df3c7eddb: Status 404 returned error can't find the container with id 57346300794b23744d58bfc9bc9b4d15329fdb33f231b0441ebde76df3c7eddb May 26 16:02:47.292755 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.292709 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" event={"ID":"5ade83d5-dc58-49a9-b12d-0ef025b0d373","Type":"ContainerStarted","Data":"9cd9eb404468565cf08cd8a1ca33e840b784ecf9f397bc9329121ed156f123b9"} May 26 16:02:47.294617 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.294585 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz" event={"ID":"27f5c521-61a5-4184-855b-243fd8577783","Type":"ContainerStarted","Data":"b7c74286d550a63d2a017933acd26f2ff7f07b2a807e42f157b6b42d6bc65066"} May 26 16:02:47.296514 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.296494 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/0.log" May 26 16:02:47.296639 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.296531 2573 generic.go:358] "Generic (PLEG): container finished" podID="a782d182-009d-4d16-babd-26586d23986b" containerID="7280afd8e3ac44500ac06face352b9f4fa3eb473f4531a9f8850f69d99e93ab5" exitCode=255 May 26 16:02:47.296639 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.296593 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-psh92" event={"ID":"a782d182-009d-4d16-babd-26586d23986b","Type":"ContainerDied","Data":"7280afd8e3ac44500ac06face352b9f4fa3eb473f4531a9f8850f69d99e93ab5"} May 26 16:02:47.296775 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.296756 2573 scope.go:117] "RemoveContainer" containerID="7280afd8e3ac44500ac06face352b9f4fa3eb473f4531a9f8850f69d99e93ab5" May 26 16:02:47.297964 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.297944 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" event={"ID":"317da9eb-b20e-413a-af49-fde829ee35d5","Type":"ContainerStarted","Data":"d9d48814584b950c8f07004f98afce0518e9eb2864fe2a9456d108d93bbd919b"} May 26 16:02:47.299188 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.299163 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s" event={"ID":"082eaf67-c961-4b94-8704-846ebd2e616e","Type":"ContainerStarted","Data":"c58277da874beca2c606a3247ffb9e5fa8d72b1ede186a9c25cd53f3025e4b73"} May 26 16:02:47.299280 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.299194 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s" event={"ID":"082eaf67-c961-4b94-8704-846ebd2e616e","Type":"ContainerStarted","Data":"57346300794b23744d58bfc9bc9b4d15329fdb33f231b0441ebde76df3c7eddb"} May 26 16:02:47.313603 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.313543 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" podStartSLOduration=2.062704175 podStartE2EDuration="5.31352816s" podCreationTimestamp="2026-05-26 16:02:42 +0000 UTC" firstStartedPulling="2026-05-26 16:02:43.189062099 +0000 UTC m=+112.048984560" lastFinishedPulling="2026-05-26 16:02:46.439886084 +0000 UTC m=+115.299808545" observedRunningTime="2026-05-26 16:02:47.312482285 +0000 UTC m=+116.172404755" watchObservedRunningTime="2026-05-26 16:02:47.31352816 +0000 UTC m=+116.173450629" May 26 16:02:47.331099 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.331056 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" podStartSLOduration=2.291287937 podStartE2EDuration="5.331042935s" podCreationTimestamp="2026-05-26 16:02:42 +0000 UTC" firstStartedPulling="2026-05-26 16:02:43.396594914 +0000 UTC m=+112.256517361" lastFinishedPulling="2026-05-26 16:02:46.436349896 +0000 UTC m=+115.296272359" observedRunningTime="2026-05-26 16:02:47.32998669 +0000 UTC m=+116.189909160" watchObservedRunningTime="2026-05-26 16:02:47.331042935 +0000 UTC m=+116.190965405" May 26 16:02:47.370698 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.370647 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-f4x4s" podStartSLOduration=2.370629314 podStartE2EDuration="2.370629314s" podCreationTimestamp="2026-05-26 16:02:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:02:47.370306982 +0000 UTC m=+116.230229451" watchObservedRunningTime="2026-05-26 16:02:47.370629314 +0000 UTC m=+116.230551786" May 26 16:02:47.385719 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:47.385670 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-flgsz" podStartSLOduration=2.193559532 podStartE2EDuration="5.385652723s" podCreationTimestamp="2026-05-26 16:02:42 +0000 UTC" firstStartedPulling="2026-05-26 16:02:43.242932259 +0000 UTC m=+112.102854719" lastFinishedPulling="2026-05-26 16:02:46.435025449 +0000 UTC m=+115.294947910" observedRunningTime="2026-05-26 16:02:47.385532176 +0000 UTC m=+116.245454645" watchObservedRunningTime="2026-05-26 16:02:47.385652723 +0000 UTC m=+116.245575173" May 26 16:02:48.304123 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.304090 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:02:48.304700 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.304485 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/0.log" May 26 16:02:48.304700 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.304523 2573 generic.go:358] "Generic (PLEG): container finished" podID="a782d182-009d-4d16-babd-26586d23986b" containerID="4951ee8b14adc28629ea5c69a02c380efb5525ab20d364828baee5140f1e72cf" exitCode=255 May 26 16:02:48.304700 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.304561 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-psh92" event={"ID":"a782d182-009d-4d16-babd-26586d23986b","Type":"ContainerDied","Data":"4951ee8b14adc28629ea5c69a02c380efb5525ab20d364828baee5140f1e72cf"} May 26 16:02:48.304700 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.304613 2573 scope.go:117] "RemoveContainer" containerID="7280afd8e3ac44500ac06face352b9f4fa3eb473f4531a9f8850f69d99e93ab5" May 26 16:02:48.304968 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.304924 2573 scope.go:117] "RemoveContainer" containerID="4951ee8b14adc28629ea5c69a02c380efb5525ab20d364828baee5140f1e72cf" May 26 16:02:48.305271 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:48.305243 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-psh92_openshift-console-operator(a782d182-009d-4d16-babd-26586d23986b)\"" pod="openshift-console-operator/console-operator-76dff48d59-psh92" podUID="a782d182-009d-4d16-babd-26586d23986b" May 26 16:02:48.921205 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.921166 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j"] May 26 16:02:48.923713 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.923686 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j" May 26 16:02:48.926915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.926887 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" May 26 16:02:48.927737 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.927713 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" May 26 16:02:48.927737 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.927727 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tm6ct\"" May 26 16:02:48.935343 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.935315 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j"] May 26 16:02:48.987050 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:48.987020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsl8\" (UniqueName: \"kubernetes.io/projected/51494e0f-df60-46c9-859c-4bb91e58078b-kube-api-access-dnsl8\") pod \"migrator-b985f5d89-lq98j\" (UID: \"51494e0f-df60-46c9-859c-4bb91e58078b\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j" May 26 16:02:49.087478 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:49.087447 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsl8\" (UniqueName: \"kubernetes.io/projected/51494e0f-df60-46c9-859c-4bb91e58078b-kube-api-access-dnsl8\") pod \"migrator-b985f5d89-lq98j\" (UID: \"51494e0f-df60-46c9-859c-4bb91e58078b\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j" May 26 16:02:49.095346 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:49.095314 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsl8\" (UniqueName: \"kubernetes.io/projected/51494e0f-df60-46c9-859c-4bb91e58078b-kube-api-access-dnsl8\") pod \"migrator-b985f5d89-lq98j\" (UID: \"51494e0f-df60-46c9-859c-4bb91e58078b\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j" May 26 16:02:49.233310 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:49.233234 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j" May 26 16:02:49.309644 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:49.309618 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:02:49.310054 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:49.310035 2573 scope.go:117] "RemoveContainer" containerID="4951ee8b14adc28629ea5c69a02c380efb5525ab20d364828baee5140f1e72cf" May 26 16:02:49.310242 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:49.310225 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-psh92_openshift-console-operator(a782d182-009d-4d16-babd-26586d23986b)\"" pod="openshift-console-operator/console-operator-76dff48d59-psh92" podUID="a782d182-009d-4d16-babd-26586d23986b" May 26 16:02:49.351124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:49.351093 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j"] May 26 16:02:49.354182 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:02:49.354156 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51494e0f_df60_46c9_859c_4bb91e58078b.slice/crio-607cf450a5cd9608661e5955b1e831cb293ec33169cf0190f86f81492e29c172 WatchSource:0}: Error finding container 607cf450a5cd9608661e5955b1e831cb293ec33169cf0190f86f81492e29c172: Status 404 returned error can't find the container with id 607cf450a5cd9608661e5955b1e831cb293ec33169cf0190f86f81492e29c172 May 26 16:02:49.796713 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:49.796682 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjm54_e636febb-62af-4bae-87f2-f5ba6ee79f71/dns-node-resolver/0.log" May 26 16:02:50.312341 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:50.312301 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j" event={"ID":"51494e0f-df60-46c9-859c-4bb91e58078b","Type":"ContainerStarted","Data":"607cf450a5cd9608661e5955b1e831cb293ec33169cf0190f86f81492e29c172"} May 26 16:02:50.497894 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:50.497853 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:50.498086 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:50.497991 2573 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 26 16:02:50.498086 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:50.498012 2573 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95767f565-hd7l2: secret "image-registry-tls" not found May 26 16:02:50.498086 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:50.498069 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls podName:8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:58.49805235 +0000 UTC m=+127.357974796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls") pod "image-registry-95767f565-hd7l2" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1") : secret "image-registry-tls" not found May 26 16:02:50.594775 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:50.594698 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b4gnl_5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a/node-ca/0.log" May 26 16:02:51.317033 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:51.316996 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j" event={"ID":"51494e0f-df60-46c9-859c-4bb91e58078b","Type":"ContainerStarted","Data":"b759448003b2eccc4d7546c56b6484e7269dddc2edf10efdaa92b06ecf21f7b8"} May 26 16:02:51.317033 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:51.317033 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j" event={"ID":"51494e0f-df60-46c9-859c-4bb91e58078b","Type":"ContainerStarted","Data":"0e9a892ad665e29b094d76cb3da106d463ad767ce6f1b9bd0b0eddee75d49041"} May 26 16:02:51.336971 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:51.336925 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-lq98j" podStartSLOduration=1.8127601119999999 podStartE2EDuration="3.336909175s" podCreationTimestamp="2026-05-26 16:02:48 +0000 UTC" firstStartedPulling="2026-05-26 16:02:49.355942776 +0000 UTC m=+118.215865224" lastFinishedPulling="2026-05-26 16:02:50.880091841 +0000 UTC m=+119.740014287" observedRunningTime="2026-05-26 16:02:51.336702754 +0000 UTC m=+120.196625225" watchObservedRunningTime="2026-05-26 16:02:51.336909175 +0000 UTC m=+120.196831644" May 26 16:02:53.031933 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:53.031901 2573 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:53.031933 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:53.031941 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:02:53.032314 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:53.032264 2573 scope.go:117] "RemoveContainer" containerID="4951ee8b14adc28629ea5c69a02c380efb5525ab20d364828baee5140f1e72cf" May 26 16:02:53.032447 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:02:53.032430 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-psh92_openshift-console-operator(a782d182-009d-4d16-babd-26586d23986b)\"" pod="openshift-console-operator/console-operator-76dff48d59-psh92" podUID="a782d182-009d-4d16-babd-26586d23986b" May 26 16:02:58.557720 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:58.557676 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:58.559958 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:58.559938 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") pod \"image-registry-95767f565-hd7l2\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:58.649229 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:58.649207 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6l5h\"" May 26 16:02:58.658148 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:58.658129 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:58.785686 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:58.785657 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-95767f565-hd7l2"] May 26 16:02:58.788790 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:02:58.788763 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecffbe8_3f45_4d4f_aaaa_c67430c55ac1.slice/crio-97d715645a2b648c34dfcd9f7de4540cc37abc055e396197c511f0c5427f0149 WatchSource:0}: Error finding container 97d715645a2b648c34dfcd9f7de4540cc37abc055e396197c511f0c5427f0149: Status 404 returned error can't find the container with id 97d715645a2b648c34dfcd9f7de4540cc37abc055e396197c511f0c5427f0149 May 26 16:02:59.341647 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:59.341615 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-95767f565-hd7l2" event={"ID":"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1","Type":"ContainerStarted","Data":"618c8d57115b4a32445b3cf107c8bf77def020514966f84aeb4803aa23270dde"} May 26 16:02:59.341647 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:59.341651 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-95767f565-hd7l2" event={"ID":"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1","Type":"ContainerStarted","Data":"97d715645a2b648c34dfcd9f7de4540cc37abc055e396197c511f0c5427f0149"} May 26 16:02:59.341881 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:59.341742 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:02:59.365649 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:02:59.365606 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-95767f565-hd7l2" podStartSLOduration=17.365593303 podStartE2EDuration="17.365593303s" podCreationTimestamp="2026-05-26 16:02:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:02:59.365191833 +0000 UTC m=+128.225114302" watchObservedRunningTime="2026-05-26 16:02:59.365593303 +0000 UTC m=+128.225515800" May 26 16:03:01.581107 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:01.581065 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:03:01.583345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:01.583321 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264cf2a2-6d0f-4220-8797-88813b9a33fe-metrics-certs\") pod \"network-metrics-daemon-vz84d\" (UID: \"264cf2a2-6d0f-4220-8797-88813b9a33fe\") " pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:03:01.886009 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:01.885930 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsz4f\"" May 26 16:03:01.893919 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:01.893897 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vz84d" May 26 16:03:02.031358 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:02.031323 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vz84d"] May 26 16:03:02.034784 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:02.034756 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264cf2a2_6d0f_4220_8797_88813b9a33fe.slice/crio-baddf58c957deffc6fa2045aa58749f064516fdf90e8d1539deb9773dbfab0ec WatchSource:0}: Error finding container baddf58c957deffc6fa2045aa58749f064516fdf90e8d1539deb9773dbfab0ec: Status 404 returned error can't find the container with id baddf58c957deffc6fa2045aa58749f064516fdf90e8d1539deb9773dbfab0ec May 26 16:03:02.350110 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:02.350074 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vz84d" event={"ID":"264cf2a2-6d0f-4220-8797-88813b9a33fe","Type":"ContainerStarted","Data":"baddf58c957deffc6fa2045aa58749f064516fdf90e8d1539deb9773dbfab0ec"} May 26 16:03:03.354111 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:03.354076 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vz84d" event={"ID":"264cf2a2-6d0f-4220-8797-88813b9a33fe","Type":"ContainerStarted","Data":"c682820d2e9502863cd6660c89164b027dd321c61ffe07e46d7e7575b9962b14"} May 26 16:03:03.354111 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:03.354116 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vz84d" event={"ID":"264cf2a2-6d0f-4220-8797-88813b9a33fe","Type":"ContainerStarted","Data":"e62f80b3a7b85264e5836936fed380b10629c5b0383309b355c9b8e02c53803f"} May 26 16:03:03.372295 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:03.372253 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vz84d" podStartSLOduration=131.453153866 podStartE2EDuration="2m12.372238024s" podCreationTimestamp="2026-05-26 16:00:51 +0000 UTC" firstStartedPulling="2026-05-26 16:03:02.036869138 +0000 UTC m=+130.896791589" lastFinishedPulling="2026-05-26 16:03:02.955953297 +0000 UTC m=+131.815875747" observedRunningTime="2026-05-26 16:03:03.371072517 +0000 UTC m=+132.230994989" watchObservedRunningTime="2026-05-26 16:03:03.372238024 +0000 UTC m=+132.232160492" May 26 16:03:03.869988 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:03.869953 2573 scope.go:117] "RemoveContainer" containerID="4951ee8b14adc28629ea5c69a02c380efb5525ab20d364828baee5140f1e72cf" May 26 16:03:04.358484 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:04.358454 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:03:04.358877 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:04.358569 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-psh92" event={"ID":"a782d182-009d-4d16-babd-26586d23986b","Type":"ContainerStarted","Data":"93d618af6af48579dc531fd71cdf6233fb4d80fbd71d76cfc26ce8334e7b9060"} May 26 16:03:04.358977 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:04.358945 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:03:04.366872 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:04.366850 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-psh92" May 26 16:03:04.377631 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:04.377586 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-psh92" podStartSLOduration=19.336009732 podStartE2EDuration="22.377572723s" podCreationTimestamp="2026-05-26 16:02:42 +0000 UTC" firstStartedPulling="2026-05-26 16:02:43.395736245 +0000 UTC m=+112.255658692" lastFinishedPulling="2026-05-26 16:02:46.437299231 +0000 UTC m=+115.297221683" observedRunningTime="2026-05-26 16:03:04.377264373 +0000 UTC m=+133.237186844" watchObservedRunningTime="2026-05-26 16:03:04.377572723 +0000 UTC m=+133.237495194" May 26 16:03:09.277414 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.277380 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj"] May 26 16:03:09.280181 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.280166 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" May 26 16:03:09.284845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.284806 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" May 26 16:03:09.284976 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.284870 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xcjtt\"" May 26 16:03:09.295492 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.295465 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj"] May 26 16:03:09.306931 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.306903 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-95767f565-hd7l2"] May 26 16:03:09.349251 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.349226 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hsw96"] May 26 16:03:09.351372 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.351356 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.353601 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.353578 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" May 26 16:03:09.353726 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.353706 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4tcs\"" May 26 16:03:09.354123 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.354102 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" May 26 16:03:09.371216 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.370166 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-pt9k8"] May 26 16:03:09.373227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.373200 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hsw96"] May 26 16:03:09.373339 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.373321 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-pt9k8" May 26 16:03:09.388240 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.388220 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" May 26 16:03:09.388714 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.388702 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" May 26 16:03:09.389432 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.389420 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6nd27\"" May 26 16:03:09.450242 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.450214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbf34c33-386f-4e41-a50d-b543f527e1ce-crio-socket\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.450399 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.450262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbf34c33-386f-4e41-a50d-b543f527e1ce-data-volume\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.450399 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.450289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbf34c33-386f-4e41-a50d-b543f527e1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.450488 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.450422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb6302e4-7513-475a-97a2-62fcbc91f37c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj\" (UID: \"eb6302e4-7513-475a-97a2-62fcbc91f37c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" May 26 16:03:09.450488 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.450469 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbf34c33-386f-4e41-a50d-b543f527e1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.450567 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.450487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdt54\" (UniqueName: \"kubernetes.io/projected/dbf34c33-386f-4e41-a50d-b543f527e1ce-kube-api-access-cdt54\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.463707 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.463682 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-pt9k8"] May 26 16:03:09.551171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.551082 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbf34c33-386f-4e41-a50d-b543f527e1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.551171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.551131 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cdt54\" (UniqueName: \"kubernetes.io/projected/dbf34c33-386f-4e41-a50d-b543f527e1ce-kube-api-access-cdt54\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.551171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.551161 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r787b\" (UniqueName: \"kubernetes.io/projected/6b080ee4-6ec6-4729-a3dd-7ae4473f41b1-kube-api-access-r787b\") pod \"downloads-57bd96b888-pt9k8\" (UID: \"6b080ee4-6ec6-4729-a3dd-7ae4473f41b1\") " pod="openshift-console/downloads-57bd96b888-pt9k8" May 26 16:03:09.551444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.551192 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbf34c33-386f-4e41-a50d-b543f527e1ce-crio-socket\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.551444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.551289 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbf34c33-386f-4e41-a50d-b543f527e1ce-crio-socket\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.551444 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.551328 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbf34c33-386f-4e41-a50d-b543f527e1ce-data-volume\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.551587 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.551441 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbf34c33-386f-4e41-a50d-b543f527e1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.551587 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.551555 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb6302e4-7513-475a-97a2-62fcbc91f37c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj\" (UID: \"eb6302e4-7513-475a-97a2-62fcbc91f37c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" May 26 16:03:09.551746 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.551722 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbf34c33-386f-4e41-a50d-b543f527e1ce-data-volume\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.552065 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.552019 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbf34c33-386f-4e41-a50d-b543f527e1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.553654 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.553630 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbf34c33-386f-4e41-a50d-b543f527e1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.554344 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.554323 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb6302e4-7513-475a-97a2-62fcbc91f37c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj\" (UID: \"eb6302e4-7513-475a-97a2-62fcbc91f37c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" May 26 16:03:09.559235 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.559212 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdt54\" (UniqueName: \"kubernetes.io/projected/dbf34c33-386f-4e41-a50d-b543f527e1ce-kube-api-access-cdt54\") pod \"insights-runtime-extractor-hsw96\" (UID: \"dbf34c33-386f-4e41-a50d-b543f527e1ce\") " pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.590032 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.589972 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" May 26 16:03:09.652633 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.652599 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r787b\" (UniqueName: \"kubernetes.io/projected/6b080ee4-6ec6-4729-a3dd-7ae4473f41b1-kube-api-access-r787b\") pod \"downloads-57bd96b888-pt9k8\" (UID: \"6b080ee4-6ec6-4729-a3dd-7ae4473f41b1\") " pod="openshift-console/downloads-57bd96b888-pt9k8" May 26 16:03:09.659665 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.659639 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hsw96" May 26 16:03:09.663864 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.663820 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r787b\" (UniqueName: \"kubernetes.io/projected/6b080ee4-6ec6-4729-a3dd-7ae4473f41b1-kube-api-access-r787b\") pod \"downloads-57bd96b888-pt9k8\" (UID: \"6b080ee4-6ec6-4729-a3dd-7ae4473f41b1\") " pod="openshift-console/downloads-57bd96b888-pt9k8" May 26 16:03:09.683412 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.681917 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-pt9k8" May 26 16:03:09.718306 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.718261 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj"] May 26 16:03:09.724785 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:09.724756 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6302e4_7513_475a_97a2_62fcbc91f37c.slice/crio-c56ae5fa0e8b9e7be31a139ad34577dadc39226eef94323ba889fcd73c878ce9 WatchSource:0}: Error finding container c56ae5fa0e8b9e7be31a139ad34577dadc39226eef94323ba889fcd73c878ce9: Status 404 returned error can't find the container with id c56ae5fa0e8b9e7be31a139ad34577dadc39226eef94323ba889fcd73c878ce9 May 26 16:03:09.810173 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.810096 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hsw96"] May 26 16:03:09.812821 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:09.812789 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf34c33_386f_4e41_a50d_b543f527e1ce.slice/crio-c027c8e6cc7eeb8dd73c40419c4485d6a228dcebfcf212446a5e5f0833f78443 WatchSource:0}: Error finding container c027c8e6cc7eeb8dd73c40419c4485d6a228dcebfcf212446a5e5f0833f78443: Status 404 returned error can't find the container with id c027c8e6cc7eeb8dd73c40419c4485d6a228dcebfcf212446a5e5f0833f78443 May 26 16:03:09.818869 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:09.818846 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-pt9k8"] May 26 16:03:09.822873 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:09.822844 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b080ee4_6ec6_4729_a3dd_7ae4473f41b1.slice/crio-03ce55509aa51c62974d450875321f7bf63310b3f01c5bd5a6f3695d19452004 WatchSource:0}: Error finding container 03ce55509aa51c62974d450875321f7bf63310b3f01c5bd5a6f3695d19452004: Status 404 returned error can't find the container with id 03ce55509aa51c62974d450875321f7bf63310b3f01c5bd5a6f3695d19452004 May 26 16:03:10.378705 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:10.378650 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-pt9k8" event={"ID":"6b080ee4-6ec6-4729-a3dd-7ae4473f41b1","Type":"ContainerStarted","Data":"03ce55509aa51c62974d450875321f7bf63310b3f01c5bd5a6f3695d19452004"} May 26 16:03:10.380361 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:10.380326 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsw96" event={"ID":"dbf34c33-386f-4e41-a50d-b543f527e1ce","Type":"ContainerStarted","Data":"88218051fcc0dafd10b30a220b8a57f5137f13bd5931cdfeb96b6db64caeb3db"} May 26 16:03:10.380361 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:10.380361 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsw96" event={"ID":"dbf34c33-386f-4e41-a50d-b543f527e1ce","Type":"ContainerStarted","Data":"c027c8e6cc7eeb8dd73c40419c4485d6a228dcebfcf212446a5e5f0833f78443"} May 26 16:03:10.381577 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:10.381530 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" event={"ID":"eb6302e4-7513-475a-97a2-62fcbc91f37c","Type":"ContainerStarted","Data":"c56ae5fa0e8b9e7be31a139ad34577dadc39226eef94323ba889fcd73c878ce9"} May 26 16:03:11.386325 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:11.386284 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsw96" event={"ID":"dbf34c33-386f-4e41-a50d-b543f527e1ce","Type":"ContainerStarted","Data":"3128f6eca1a5e8485d706ed2b38f98338f22defb3cb73c47260fea2e06585317"} May 26 16:03:11.387810 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:11.387781 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" event={"ID":"eb6302e4-7513-475a-97a2-62fcbc91f37c","Type":"ContainerStarted","Data":"29871f410eef8ad8efda3954cd463796c684db390c32b09a8df9359fe32a6d68"} May 26 16:03:11.388111 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:11.388087 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" May 26 16:03:11.394030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:11.393988 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" May 26 16:03:11.408438 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:11.408390 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj" podStartSLOduration=1.218959596 podStartE2EDuration="2.408373536s" podCreationTimestamp="2026-05-26 16:03:09 +0000 UTC" firstStartedPulling="2026-05-26 16:03:09.727582853 +0000 UTC m=+138.587505317" lastFinishedPulling="2026-05-26 16:03:10.916996796 +0000 UTC m=+139.776919257" observedRunningTime="2026-05-26 16:03:11.406336593 +0000 UTC m=+140.266259065" watchObservedRunningTime="2026-05-26 16:03:11.408373536 +0000 UTC m=+140.268296005" May 26 16:03:12.393171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.393128 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsw96" event={"ID":"dbf34c33-386f-4e41-a50d-b543f527e1ce","Type":"ContainerStarted","Data":"aa92ce45ae6a2e50dab59288a6afb4cc32235b16f268876d0159eac1f83640bc"} May 26 16:03:12.412918 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.412872 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hsw96" podStartSLOduration=0.979830227 podStartE2EDuration="3.412854117s" podCreationTimestamp="2026-05-26 16:03:09 +0000 UTC" firstStartedPulling="2026-05-26 16:03:09.86142857 +0000 UTC m=+138.721351020" lastFinishedPulling="2026-05-26 16:03:12.294452449 +0000 UTC m=+141.154374910" observedRunningTime="2026-05-26 16:03:12.411488673 +0000 UTC m=+141.271411144" watchObservedRunningTime="2026-05-26 16:03:12.412854117 +0000 UTC m=+141.272776587" May 26 16:03:12.424078 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.424049 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-42lfv"] May 26 16:03:12.426119 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.426092 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.429987 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.429959 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" May 26 16:03:12.430120 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.429994 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" May 26 16:03:12.430120 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.430012 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-579jh\"" May 26 16:03:12.430120 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.430028 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" May 26 16:03:12.430120 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.429999 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" May 26 16:03:12.430120 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.429971 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" May 26 16:03:12.438189 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.438167 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-42lfv"] May 26 16:03:12.473926 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.473900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e72ca05-4dae-4954-b79c-f9c34a00ed13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.474048 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.473941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e72ca05-4dae-4954-b79c-f9c34a00ed13-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.474048 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.473984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e72ca05-4dae-4954-b79c-f9c34a00ed13-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.474193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.474168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqxg9\" (UniqueName: \"kubernetes.io/projected/3e72ca05-4dae-4954-b79c-f9c34a00ed13-kube-api-access-bqxg9\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.575227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.575135 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxg9\" (UniqueName: \"kubernetes.io/projected/3e72ca05-4dae-4954-b79c-f9c34a00ed13-kube-api-access-bqxg9\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.575227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.575209 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e72ca05-4dae-4954-b79c-f9c34a00ed13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.575454 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.575242 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e72ca05-4dae-4954-b79c-f9c34a00ed13-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.575454 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.575268 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e72ca05-4dae-4954-b79c-f9c34a00ed13-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.575454 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:12.575427 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found May 26 16:03:12.575618 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:12.575497 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e72ca05-4dae-4954-b79c-f9c34a00ed13-prometheus-operator-tls podName:3e72ca05-4dae-4954-b79c-f9c34a00ed13 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:13.075481212 +0000 UTC m=+141.935403682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/3e72ca05-4dae-4954-b79c-f9c34a00ed13-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-42lfv" (UID: "3e72ca05-4dae-4954-b79c-f9c34a00ed13") : secret "prometheus-operator-tls" not found May 26 16:03:12.576049 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.576028 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e72ca05-4dae-4954-b79c-f9c34a00ed13-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.578266 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.578245 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e72ca05-4dae-4954-b79c-f9c34a00ed13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:12.586734 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:12.586713 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqxg9\" (UniqueName: \"kubernetes.io/projected/3e72ca05-4dae-4954-b79c-f9c34a00ed13-kube-api-access-bqxg9\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:13.080415 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:13.080377 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e72ca05-4dae-4954-b79c-f9c34a00ed13-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:13.083137 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:13.083106 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e72ca05-4dae-4954-b79c-f9c34a00ed13-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-42lfv\" (UID: \"3e72ca05-4dae-4954-b79c-f9c34a00ed13\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:13.337939 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:13.337856 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" May 26 16:03:13.474927 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:13.474891 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-42lfv"] May 26 16:03:13.478429 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:13.478402 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e72ca05_4dae_4954_b79c_f9c34a00ed13.slice/crio-d58a7424ef3bd39f72de44fd405a44d00d1a9def709c22819bed47daf9fd7401 WatchSource:0}: Error finding container d58a7424ef3bd39f72de44fd405a44d00d1a9def709c22819bed47daf9fd7401: Status 404 returned error can't find the container with id d58a7424ef3bd39f72de44fd405a44d00d1a9def709c22819bed47daf9fd7401 May 26 16:03:14.400903 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:14.400863 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" event={"ID":"3e72ca05-4dae-4954-b79c-f9c34a00ed13","Type":"ContainerStarted","Data":"d58a7424ef3bd39f72de44fd405a44d00d1a9def709c22819bed47daf9fd7401"} May 26 16:03:15.404596 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:15.404558 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" event={"ID":"3e72ca05-4dae-4954-b79c-f9c34a00ed13","Type":"ContainerStarted","Data":"e7f78d2185aef41b3291c579b7b4e5f4be99970e5c39ce33d724687f124c4dd7"} May 26 16:03:15.404596 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:15.404593 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" event={"ID":"3e72ca05-4dae-4954-b79c-f9c34a00ed13","Type":"ContainerStarted","Data":"3845c5904893331a7fb984eea7e1edceeb2662d2dc097e7e86e05575c2b4521d"} May 26 16:03:15.422686 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:15.422645 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-42lfv" podStartSLOduration=2.075596402 podStartE2EDuration="3.422631853s" podCreationTimestamp="2026-05-26 16:03:12 +0000 UTC" firstStartedPulling="2026-05-26 16:03:13.480344065 +0000 UTC m=+142.340266512" lastFinishedPulling="2026-05-26 16:03:14.827379492 +0000 UTC m=+143.687301963" observedRunningTime="2026-05-26 16:03:15.421184508 +0000 UTC m=+144.281106976" watchObservedRunningTime="2026-05-26 16:03:15.422631853 +0000 UTC m=+144.282554322" May 26 16:03:17.778504 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.778462 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-4qch8"] May 26 16:03:17.781142 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.781122 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.783640 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.783618 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" May 26 16:03:17.783640 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.783632 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" May 26 16:03:17.784076 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.784056 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f2tpm\"" May 26 16:03:17.792190 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.792169 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-4qch8"] May 26 16:03:17.817490 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.817467 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dctfv"] May 26 16:03:17.819624 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.819611 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.820027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.819998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.820141 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.820067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.820218 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.820117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nqn\" (UniqueName: \"kubernetes.io/projected/a315e186-fcad-4ae6-adf5-74b3a6e5903b-kube-api-access-n7nqn\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.820271 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.820222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a315e186-fcad-4ae6-adf5-74b3a6e5903b-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.822233 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.822172 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" May 26 16:03:17.822233 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.822200 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" May 26 16:03:17.822369 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.822261 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cb4pg\"" May 26 16:03:17.823017 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.823000 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" May 26 16:03:17.920716 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.920683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-textfile\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.920890 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.920741 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.920890 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.920770 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aceee1e1-ee32-492f-a009-d97fb515b672-root\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.920890 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.920798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-tls\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.920890 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.920858 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nqn\" (UniqueName: \"kubernetes.io/projected/a315e186-fcad-4ae6-adf5-74b3a6e5903b-kube-api-access-n7nqn\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.920890 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:17.920881 2573 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found May 26 16:03:17.921137 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.920923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aceee1e1-ee32-492f-a009-d97fb515b672-metrics-client-ca\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.921137 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:17.920950 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-tls podName:a315e186-fcad-4ae6-adf5-74b3a6e5903b nodeName:}" failed. No retries permitted until 2026-05-26 16:03:18.420929766 +0000 UTC m=+147.280852213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-4qch8" (UID: "a315e186-fcad-4ae6-adf5-74b3a6e5903b") : secret "openshift-state-metrics-tls" not found May 26 16:03:17.921274 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.921206 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7hh\" (UniqueName: \"kubernetes.io/projected/aceee1e1-ee32-492f-a009-d97fb515b672-kube-api-access-6z7hh\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.921439 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.921410 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a315e186-fcad-4ae6-adf5-74b3a6e5903b-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.921554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.921455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.921554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.921498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aceee1e1-ee32-492f-a009-d97fb515b672-sys\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.921554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.921536 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-wtmp\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.921699 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.921570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-accelerators-collector-config\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:17.925877 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.921819 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.925877 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.922373 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a315e186-fcad-4ae6-adf5-74b3a6e5903b-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.925877 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.924464 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:17.932122 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:17.932098 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nqn\" (UniqueName: \"kubernetes.io/projected/a315e186-fcad-4ae6-adf5-74b3a6e5903b-kube-api-access-n7nqn\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:18.023267 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023235 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-accelerators-collector-config\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023431 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023320 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-textfile\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023431 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023365 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aceee1e1-ee32-492f-a009-d97fb515b672-root\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023431 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023395 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-tls\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023580 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023439 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aceee1e1-ee32-492f-a009-d97fb515b672-metrics-client-ca\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023580 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023463 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7hh\" (UniqueName: \"kubernetes.io/projected/aceee1e1-ee32-492f-a009-d97fb515b672-kube-api-access-6z7hh\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023580 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023467 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aceee1e1-ee32-492f-a009-d97fb515b672-root\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023580 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023504 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023580 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023534 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aceee1e1-ee32-492f-a009-d97fb515b672-sys\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023580 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023565 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-wtmp\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023869 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:18.023612 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found May 26 16:03:18.023869 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:18.023683 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-tls podName:aceee1e1-ee32-492f-a009-d97fb515b672 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:18.523662816 +0000 UTC m=+147.383585270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-tls") pod "node-exporter-dctfv" (UID: "aceee1e1-ee32-492f-a009-d97fb515b672") : secret "node-exporter-tls" not found May 26 16:03:18.023869 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023687 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-textfile\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023869 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023712 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-wtmp\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.023869 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023764 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aceee1e1-ee32-492f-a009-d97fb515b672-sys\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.024138 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023905 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-accelerators-collector-config\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.024138 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.023976 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aceee1e1-ee32-492f-a009-d97fb515b672-metrics-client-ca\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.026333 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.026308 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.034243 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.034189 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7hh\" (UniqueName: \"kubernetes.io/projected/aceee1e1-ee32-492f-a009-d97fb515b672-kube-api-access-6z7hh\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.427952 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.427864 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:18.428102 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:18.428009 2573 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found May 26 16:03:18.428164 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:18.428113 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-tls podName:a315e186-fcad-4ae6-adf5-74b3a6e5903b nodeName:}" failed. No retries permitted until 2026-05-26 16:03:19.42809195 +0000 UTC m=+148.288014397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-4qch8" (UID: "a315e186-fcad-4ae6-adf5-74b3a6e5903b") : secret "openshift-state-metrics-tls" not found May 26 16:03:18.529305 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.529267 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-tls\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.532155 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.532127 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aceee1e1-ee32-492f-a009-d97fb515b672-node-exporter-tls\") pod \"node-exporter-dctfv\" (UID: \"aceee1e1-ee32-492f-a009-d97fb515b672\") " pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.728933 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:18.728858 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dctfv" May 26 16:03:18.738670 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:18.738637 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaceee1e1_ee32_492f_a009_d97fb515b672.slice/crio-5ade2466396ed6bb81858c5353f0cb384d2b2a9b0172d6b0bfeaf8802b82abbf WatchSource:0}: Error finding container 5ade2466396ed6bb81858c5353f0cb384d2b2a9b0172d6b0bfeaf8802b82abbf: Status 404 returned error can't find the container with id 5ade2466396ed6bb81858c5353f0cb384d2b2a9b0172d6b0bfeaf8802b82abbf May 26 16:03:19.312283 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.312240 2573 patch_prober.go:28] interesting pod/image-registry-95767f565-hd7l2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} May 26 16:03:19.312653 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.312295 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-95767f565-hd7l2" podUID="8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 26 16:03:19.417717 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.417662 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dctfv" event={"ID":"aceee1e1-ee32-492f-a009-d97fb515b672","Type":"ContainerStarted","Data":"5ade2466396ed6bb81858c5353f0cb384d2b2a9b0172d6b0bfeaf8802b82abbf"} May 26 16:03:19.438148 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.438110 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:19.441307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.441274 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a315e186-fcad-4ae6-adf5-74b3a6e5903b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-4qch8\" (UID: \"a315e186-fcad-4ae6-adf5-74b3a6e5903b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:19.592570 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.592490 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" May 26 16:03:19.725396 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.725361 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-4qch8"] May 26 16:03:19.741607 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:19.741554 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda315e186_fcad_4ae6_adf5_74b3a6e5903b.slice/crio-ac8a18bf2c5c19933aeb2dc108f990994e936d1f8d031851f4340bca4afe4d2f WatchSource:0}: Error finding container ac8a18bf2c5c19933aeb2dc108f990994e936d1f8d031851f4340bca4afe4d2f: Status 404 returned error can't find the container with id ac8a18bf2c5c19933aeb2dc108f990994e936d1f8d031851f4340bca4afe4d2f May 26 16:03:19.888511 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.888455 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c47c44689-7g8p2"] May 26 16:03:19.899607 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.899582 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:19.904464 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.903433 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" May 26 16:03:19.904581 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.904528 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" May 26 16:03:19.905372 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.904740 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" May 26 16:03:19.905372 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.904863 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c47c44689-7g8p2"] May 26 16:03:19.905372 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.904893 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" May 26 16:03:19.905372 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.905027 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" May 26 16:03:19.905372 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.905056 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vw4r9\"" May 26 16:03:19.908008 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.907424 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-79afs0t5u0rv6\"" May 26 16:03:19.942668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.942643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-tls\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:19.942784 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.942686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5qcb\" (UniqueName: \"kubernetes.io/projected/04523e11-5c21-4b16-aea5-ba8ed8999e39-kube-api-access-f5qcb\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:19.942784 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.942769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:19.942911 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.942843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-grpc-tls\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:19.942911 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.942896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:19.943015 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.942933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:19.943015 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.942975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04523e11-5c21-4b16-aea5-ba8ed8999e39-metrics-client-ca\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:19.943116 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:19.943030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.044360 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.044321 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.044535 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.044370 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.044535 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.044414 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04523e11-5c21-4b16-aea5-ba8ed8999e39-metrics-client-ca\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.044535 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.044452 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.044535 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.044496 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-tls\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.044535 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.044523 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f5qcb\" (UniqueName: \"kubernetes.io/projected/04523e11-5c21-4b16-aea5-ba8ed8999e39-kube-api-access-f5qcb\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.044786 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.044571 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.044786 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.044662 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-grpc-tls\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.045684 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.045634 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04523e11-5c21-4b16-aea5-ba8ed8999e39-metrics-client-ca\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.047477 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.047435 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.047646 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.047615 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.047743 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.047721 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-grpc-tls\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.048259 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.048235 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.048349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.048329 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-tls\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.048426 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.048404 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04523e11-5c21-4b16-aea5-ba8ed8999e39-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.061768 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.061745 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5qcb\" (UniqueName: \"kubernetes.io/projected/04523e11-5c21-4b16-aea5-ba8ed8999e39-kube-api-access-f5qcb\") pod \"thanos-querier-7c47c44689-7g8p2\" (UID: \"04523e11-5c21-4b16-aea5-ba8ed8999e39\") " pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.076542 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.076506 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cbfc9dc65-fzqqq"] May 26 16:03:20.098399 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.098332 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cbfc9dc65-fzqqq"] May 26 16:03:20.098529 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.098464 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.101319 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.101296 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" May 26 16:03:20.101442 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.101296 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" May 26 16:03:20.101717 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.101694 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" May 26 16:03:20.102314 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.102295 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" May 26 16:03:20.102430 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.102338 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" May 26 16:03:20.102783 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.102764 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ssfl9\"" May 26 16:03:20.106199 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.106108 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" May 26 16:03:20.146020 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.145979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-oauth-serving-cert\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.146146 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.146054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnhj9\" (UniqueName: \"kubernetes.io/projected/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-kube-api-access-jnhj9\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.146146 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.146101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-serving-cert\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.146554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.146302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-service-ca\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.146554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.146383 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-oauth-config\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.146554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.146409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-trusted-ca-bundle\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.146554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.146464 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-config\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.215326 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.215292 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:20.247765 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.247728 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-oauth-config\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.247938 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.247775 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-trusted-ca-bundle\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.247938 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.247840 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-config\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.247938 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.247880 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-oauth-serving-cert\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.247938 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.247909 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jnhj9\" (UniqueName: \"kubernetes.io/projected/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-kube-api-access-jnhj9\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.247938 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.247932 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-serving-cert\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.248220 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.247990 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-service-ca\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.249048 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.248980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-service-ca\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.249048 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.248980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-oauth-serving-cert\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.249206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.249130 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-trusted-ca-bundle\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.249206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.249148 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-config\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.250710 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.250669 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-oauth-config\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.250864 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.250844 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-serving-cert\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.258536 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.258514 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnhj9\" (UniqueName: \"kubernetes.io/projected/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-kube-api-access-jnhj9\") pod \"console-5cbfc9dc65-fzqqq\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.367799 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.367741 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c47c44689-7g8p2"] May 26 16:03:20.370154 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:20.370122 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04523e11_5c21_4b16_aea5_ba8ed8999e39.slice/crio-1500413ff69ef31ee6d29d14236fbdb3da321edee74f32dd04454d7a822c57a9 WatchSource:0}: Error finding container 1500413ff69ef31ee6d29d14236fbdb3da321edee74f32dd04454d7a822c57a9: Status 404 returned error can't find the container with id 1500413ff69ef31ee6d29d14236fbdb3da321edee74f32dd04454d7a822c57a9 May 26 16:03:20.410825 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.410785 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:20.423465 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.423423 2573 generic.go:358] "Generic (PLEG): container finished" podID="aceee1e1-ee32-492f-a009-d97fb515b672" containerID="e2b9bc32520befceaabae9c86df6dfddcd5361632e0d58cd7570639739761df8" exitCode=0 May 26 16:03:20.423573 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.423504 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dctfv" event={"ID":"aceee1e1-ee32-492f-a009-d97fb515b672","Type":"ContainerDied","Data":"e2b9bc32520befceaabae9c86df6dfddcd5361632e0d58cd7570639739761df8"} May 26 16:03:20.425572 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.425546 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" event={"ID":"a315e186-fcad-4ae6-adf5-74b3a6e5903b","Type":"ContainerStarted","Data":"073f96001360e023c4ae7b018636378958a41fa211bdc49d187fea3b159d444c"} May 26 16:03:20.425679 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.425579 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" event={"ID":"a315e186-fcad-4ae6-adf5-74b3a6e5903b","Type":"ContainerStarted","Data":"9208947450a3b17239ee97ec7a305bdbd161a5058fb2b2f5deeed414c1ae165a"} May 26 16:03:20.425679 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.425593 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" event={"ID":"a315e186-fcad-4ae6-adf5-74b3a6e5903b","Type":"ContainerStarted","Data":"ac8a18bf2c5c19933aeb2dc108f990994e936d1f8d031851f4340bca4afe4d2f"} May 26 16:03:20.426941 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.426865 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" event={"ID":"04523e11-5c21-4b16-aea5-ba8ed8999e39","Type":"ContainerStarted","Data":"1500413ff69ef31ee6d29d14236fbdb3da321edee74f32dd04454d7a822c57a9"} May 26 16:03:20.552979 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:20.552947 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cbfc9dc65-fzqqq"] May 26 16:03:20.556224 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:20.556193 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3cc2ff_246f_48cf_88ce_338b1eb2c4a0.slice/crio-771a0254fb558cdee2d47352978899ab75278085a89f2cc15627a18cf0a87f01 WatchSource:0}: Error finding container 771a0254fb558cdee2d47352978899ab75278085a89f2cc15627a18cf0a87f01: Status 404 returned error can't find the container with id 771a0254fb558cdee2d47352978899ab75278085a89f2cc15627a18cf0a87f01 May 26 16:03:21.432063 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:21.432023 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbfc9dc65-fzqqq" event={"ID":"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0","Type":"ContainerStarted","Data":"771a0254fb558cdee2d47352978899ab75278085a89f2cc15627a18cf0a87f01"} May 26 16:03:21.434688 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:21.434632 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dctfv" event={"ID":"aceee1e1-ee32-492f-a009-d97fb515b672","Type":"ContainerStarted","Data":"a8fe14edb39faef3de71b6266c23bb989992428e1818e33ccacb57d08c47e482"} May 26 16:03:21.434688 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:21.434671 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dctfv" event={"ID":"aceee1e1-ee32-492f-a009-d97fb515b672","Type":"ContainerStarted","Data":"11ab620cbcbde9dc4c565dca5db641bec1ee2312ddaaf62fabe97cb7f628e53d"} May 26 16:03:21.893397 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:21.893337 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dctfv" podStartSLOduration=3.878122854 podStartE2EDuration="4.893318317s" podCreationTimestamp="2026-05-26 16:03:17 +0000 UTC" firstStartedPulling="2026-05-26 16:03:18.740923951 +0000 UTC m=+147.600846400" lastFinishedPulling="2026-05-26 16:03:19.756119416 +0000 UTC m=+148.616041863" observedRunningTime="2026-05-26 16:03:21.455570531 +0000 UTC m=+150.315493001" watchObservedRunningTime="2026-05-26 16:03:21.893318317 +0000 UTC m=+150.753240790" May 26 16:03:22.569595 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:22.569560 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-kqb24"] May 26 16:03:22.571982 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:22.571960 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" May 26 16:03:22.574363 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:22.574338 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" May 26 16:03:22.574483 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:22.574400 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-hszck\"" May 26 16:03:22.584770 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:22.584747 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-kqb24"] May 26 16:03:22.673241 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:22.673203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3211cf-6726-4e08-b6be-a85fbd33eb2c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-kqb24\" (UID: \"0e3211cf-6726-4e08-b6be-a85fbd33eb2c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" May 26 16:03:22.774782 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:22.774742 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3211cf-6726-4e08-b6be-a85fbd33eb2c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-kqb24\" (UID: \"0e3211cf-6726-4e08-b6be-a85fbd33eb2c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" May 26 16:03:22.774990 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:22.774918 2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found May 26 16:03:22.775054 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:22.775001 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e3211cf-6726-4e08-b6be-a85fbd33eb2c-monitoring-plugin-cert podName:0e3211cf-6726-4e08-b6be-a85fbd33eb2c nodeName:}" failed. No retries permitted until 2026-05-26 16:03:23.27497805 +0000 UTC m=+152.134900514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0e3211cf-6726-4e08-b6be-a85fbd33eb2c-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-kqb24" (UID: "0e3211cf-6726-4e08-b6be-a85fbd33eb2c") : secret "monitoring-plugin-cert" not found May 26 16:03:23.029470 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.029370 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8548648857-6pkmp"] May 26 16:03:23.034053 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.032241 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.034825 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.034789 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ddn2f\"" May 26 16:03:23.035290 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.035263 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" May 26 16:03:23.035290 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.035283 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" May 26 16:03:23.035615 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.035597 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" May 26 16:03:23.035715 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.035686 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" May 26 16:03:23.036094 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.036073 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" May 26 16:03:23.045489 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.045466 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" May 26 16:03:23.046365 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.046345 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8548648857-6pkmp"] May 26 16:03:23.078717 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.078684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-metrics-client-ca\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.078888 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.078803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-telemeter-client-tls\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.078888 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.078870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.078994 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.078922 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.078994 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.078964 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwgqm\" (UniqueName: \"kubernetes.io/projected/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-kube-api-access-vwgqm\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.079066 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.078996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-secret-telemeter-client\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.079066 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.079044 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-federate-client-tls\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.079127 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.079080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-serving-certs-ca-bundle\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.180404 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.180353 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwgqm\" (UniqueName: \"kubernetes.io/projected/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-kube-api-access-vwgqm\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.180404 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.180410 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-secret-telemeter-client\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.180651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.180466 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-federate-client-tls\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.180651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.180498 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-serving-certs-ca-bundle\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.180651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.180566 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-metrics-client-ca\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.180651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.180630 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-telemeter-client-tls\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.180880 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.180658 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.180880 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.180706 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.181390 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.181359 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-serving-certs-ca-bundle\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.181528 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.181413 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-metrics-client-ca\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.181782 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.181756 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.183710 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.183686 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-federate-client-tls\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.183802 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.183730 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.183802 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.183755 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-secret-telemeter-client\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.184452 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.184431 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-telemeter-client-tls\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.188897 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.188872 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwgqm\" (UniqueName: \"kubernetes.io/projected/0aa5eec1-ca38-4545-a077-e6c873b4c7c5-kube-api-access-vwgqm\") pod \"telemeter-client-8548648857-6pkmp\" (UID: \"0aa5eec1-ca38-4545-a077-e6c873b4c7c5\") " pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.282235 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.282136 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3211cf-6726-4e08-b6be-a85fbd33eb2c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-kqb24\" (UID: \"0e3211cf-6726-4e08-b6be-a85fbd33eb2c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" May 26 16:03:23.284925 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.284902 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3211cf-6726-4e08-b6be-a85fbd33eb2c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-kqb24\" (UID: \"0e3211cf-6726-4e08-b6be-a85fbd33eb2c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" May 26 16:03:23.344786 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.344748 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" May 26 16:03:23.487054 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:23.487016 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" May 26 16:03:25.525452 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.523787 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dcfd8bd6f-ffgl7"] May 26 16:03:25.527800 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.527374 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.543434 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.543404 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dcfd8bd6f-ffgl7"] May 26 16:03:25.605034 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.604996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-serving-cert\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.605227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.605061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lwb\" (UniqueName: \"kubernetes.io/projected/22adef2c-3729-488a-a440-b1bd060dd355-kube-api-access-p5lwb\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.605227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.605114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-oauth-serving-cert\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.605227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.605202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-console-config\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.605362 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.605256 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-oauth-config\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.605362 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.605335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-trusted-ca-bundle\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.605453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.605374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-service-ca\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.706272 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.706222 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-console-config\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.706466 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.706292 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-oauth-config\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.706466 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.706351 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-trusted-ca-bundle\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.706466 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.706391 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-service-ca\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.706466 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.706431 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-serving-cert\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.706719 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.706468 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lwb\" (UniqueName: \"kubernetes.io/projected/22adef2c-3729-488a-a440-b1bd060dd355-kube-api-access-p5lwb\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.706719 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.706507 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-oauth-serving-cert\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.707135 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.707085 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-console-config\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.707252 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.707201 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-oauth-serving-cert\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.707368 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.707344 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-service-ca\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.707368 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.707357 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-trusted-ca-bundle\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.709152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.709128 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-serving-cert\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.709229 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.709190 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-oauth-config\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.716594 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.716570 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lwb\" (UniqueName: \"kubernetes.io/projected/22adef2c-3729-488a-a440-b1bd060dd355-kube-api-access-p5lwb\") pod \"console-7dcfd8bd6f-ffgl7\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:25.844442 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:25.844346 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:28.540570 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:28.540535 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-p8lq6" podUID="c42e8c71-312c-41a0-a8d9-b116879302ae" May 26 16:03:28.554686 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:03:28.554649 2573 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-rs5nq" podUID="ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a" May 26 16:03:29.312595 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:29.312549 2573 patch_prober.go:28] interesting pod/image-registry-95767f565-hd7l2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} May 26 16:03:29.312761 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:29.312604 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-95767f565-hd7l2" podUID="8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 26 16:03:29.462382 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:29.462351 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:03:29.462606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:29.462359 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8lq6" May 26 16:03:30.472490 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:30.470743 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" event={"ID":"a315e186-fcad-4ae6-adf5-74b3a6e5903b","Type":"ContainerStarted","Data":"0e96bdb154f68dfdabf92f125c0196c9d320efa1d8917903d3f910f599655d1d"} May 26 16:03:30.491973 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:30.491883 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-4qch8" podStartSLOduration=3.127470525 podStartE2EDuration="13.491863265s" podCreationTimestamp="2026-05-26 16:03:17 +0000 UTC" firstStartedPulling="2026-05-26 16:03:19.955601027 +0000 UTC m=+148.815523490" lastFinishedPulling="2026-05-26 16:03:30.319993769 +0000 UTC m=+159.179916230" observedRunningTime="2026-05-26 16:03:30.489665233 +0000 UTC m=+159.349587704" watchObservedRunningTime="2026-05-26 16:03:30.491863265 +0000 UTC m=+159.351785735" May 26 16:03:30.520113 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:30.519885 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-kqb24"] May 26 16:03:30.539822 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:30.539771 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dcfd8bd6f-ffgl7"] May 26 16:03:30.543086 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:30.543060 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22adef2c_3729_488a_a440_b1bd060dd355.slice/crio-b6dd228a2888a6ee34df1330d20cba6f5ebb2f8ea848f24f9cb217dff09906e9 WatchSource:0}: Error finding container b6dd228a2888a6ee34df1330d20cba6f5ebb2f8ea848f24f9cb217dff09906e9: Status 404 returned error can't find the container with id b6dd228a2888a6ee34df1330d20cba6f5ebb2f8ea848f24f9cb217dff09906e9 May 26 16:03:30.559941 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:30.559791 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8548648857-6pkmp"] May 26 16:03:30.562250 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:30.562224 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa5eec1_ca38_4545_a077_e6c873b4c7c5.slice/crio-7cf83b42fd6e4b057cd485c564e6ec6dab38f2e42012d6867fd342f486147d5c WatchSource:0}: Error finding container 7cf83b42fd6e4b057cd485c564e6ec6dab38f2e42012d6867fd342f486147d5c: Status 404 returned error can't find the container with id 7cf83b42fd6e4b057cd485c564e6ec6dab38f2e42012d6867fd342f486147d5c May 26 16:03:31.486809 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.486760 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbfc9dc65-fzqqq" event={"ID":"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0","Type":"ContainerStarted","Data":"9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf"} May 26 16:03:31.489957 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.489890 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcfd8bd6f-ffgl7" event={"ID":"22adef2c-3729-488a-a440-b1bd060dd355","Type":"ContainerStarted","Data":"3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b"} May 26 16:03:31.489957 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.489924 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcfd8bd6f-ffgl7" event={"ID":"22adef2c-3729-488a-a440-b1bd060dd355","Type":"ContainerStarted","Data":"b6dd228a2888a6ee34df1330d20cba6f5ebb2f8ea848f24f9cb217dff09906e9"} May 26 16:03:31.495007 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.494820 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-pt9k8" event={"ID":"6b080ee4-6ec6-4729-a3dd-7ae4473f41b1","Type":"ContainerStarted","Data":"59414f02b139b01a27a26d09b0e147b173c9e8e259e2f00d91836c272184c1bf"} May 26 16:03:31.495460 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.495316 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-pt9k8" May 26 16:03:31.497994 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.497951 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" event={"ID":"0aa5eec1-ca38-4545-a077-e6c873b4c7c5","Type":"ContainerStarted","Data":"7cf83b42fd6e4b057cd485c564e6ec6dab38f2e42012d6867fd342f486147d5c"} May 26 16:03:31.505316 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.505286 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" event={"ID":"04523e11-5c21-4b16-aea5-ba8ed8999e39","Type":"ContainerStarted","Data":"74a280658540f8c8ad5b340b5f0fd4b7e9503abb2f3d0e3bab583531fbbacd25"} May 26 16:03:31.505458 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.505321 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" event={"ID":"04523e11-5c21-4b16-aea5-ba8ed8999e39","Type":"ContainerStarted","Data":"322f8b9c6f349900c03e3d9e8a7a54b987e35558b3179039419ff7ec4216f10d"} May 26 16:03:31.505458 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.505336 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" event={"ID":"04523e11-5c21-4b16-aea5-ba8ed8999e39","Type":"ContainerStarted","Data":"f58996870a2c5b39dee2059d77c444276f48e0dffc501faf8060433d0a3a5d9c"} May 26 16:03:31.507799 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.507777 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-pt9k8" May 26 16:03:31.509238 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.508675 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" event={"ID":"0e3211cf-6726-4e08-b6be-a85fbd33eb2c","Type":"ContainerStarted","Data":"24efb356ac6dda06fff4f8183cc7a4bc463032c41dd5f0ddabfcad769a41b8e2"} May 26 16:03:31.513783 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.513731 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5cbfc9dc65-fzqqq" podStartSLOduration=1.732222621 podStartE2EDuration="11.513716254s" podCreationTimestamp="2026-05-26 16:03:20 +0000 UTC" firstStartedPulling="2026-05-26 16:03:20.558287783 +0000 UTC m=+149.418210230" lastFinishedPulling="2026-05-26 16:03:30.339781413 +0000 UTC m=+159.199703863" observedRunningTime="2026-05-26 16:03:31.509076806 +0000 UTC m=+160.368999276" watchObservedRunningTime="2026-05-26 16:03:31.513716254 +0000 UTC m=+160.373638724" May 26 16:03:31.528606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.528556 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7dcfd8bd6f-ffgl7" podStartSLOduration=6.528540973 podStartE2EDuration="6.528540973s" podCreationTimestamp="2026-05-26 16:03:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:03:31.527227371 +0000 UTC m=+160.387149843" watchObservedRunningTime="2026-05-26 16:03:31.528540973 +0000 UTC m=+160.388463443" May 26 16:03:31.545881 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:31.545812 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-pt9k8" podStartSLOduration=1.999242142 podStartE2EDuration="22.545796139s" podCreationTimestamp="2026-05-26 16:03:09 +0000 UTC" firstStartedPulling="2026-05-26 16:03:09.82498542 +0000 UTC m=+138.684907867" lastFinishedPulling="2026-05-26 16:03:30.371539411 +0000 UTC m=+159.231461864" observedRunningTime="2026-05-26 16:03:31.543745323 +0000 UTC m=+160.403667794" watchObservedRunningTime="2026-05-26 16:03:31.545796139 +0000 UTC m=+160.405718605" May 26 16:03:32.514545 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:32.514449 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" event={"ID":"04523e11-5c21-4b16-aea5-ba8ed8999e39","Type":"ContainerStarted","Data":"194c7787c2c5e366bffe7d2187b1500fec5a50150a75643be0cd71b944d5c43d"} May 26 16:03:33.485562 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.485524 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:03:33.485760 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.485627 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:03:33.488131 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.488101 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e8c71-312c-41a0-a8d9-b116879302ae-metrics-tls\") pod \"dns-default-p8lq6\" (UID: \"c42e8c71-312c-41a0-a8d9-b116879302ae\") " pod="openshift-dns/dns-default-p8lq6" May 26 16:03:33.488271 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.488226 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a-cert\") pod \"ingress-canary-rs5nq\" (UID: \"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a\") " pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:03:33.520753 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.520686 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" event={"ID":"04523e11-5c21-4b16-aea5-ba8ed8999e39","Type":"ContainerStarted","Data":"55a7d3d4bae57a549977327c8436ea37619cbe7ba62b2637cc88c5ec237e5be6"} May 26 16:03:33.520753 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.520729 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" event={"ID":"04523e11-5c21-4b16-aea5-ba8ed8999e39","Type":"ContainerStarted","Data":"bf9eff4ea0e40ec6616128e57bc73baad27c335f9eceab5805c34cfc8a2f8f27"} May 26 16:03:33.521212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.520960 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:33.522309 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.522283 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" event={"ID":"0e3211cf-6726-4e08-b6be-a85fbd33eb2c","Type":"ContainerStarted","Data":"b8793da329bb45d07f6189dcbd2ef538c21e628d75da1dacff0edcbf8c439507"} May 26 16:03:33.522566 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.522542 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" May 26 16:03:33.527962 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.527944 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" May 26 16:03:33.545302 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.545256 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" podStartSLOduration=2.539254965 podStartE2EDuration="14.545240777s" podCreationTimestamp="2026-05-26 16:03:19 +0000 UTC" firstStartedPulling="2026-05-26 16:03:20.372134486 +0000 UTC m=+149.232056933" lastFinishedPulling="2026-05-26 16:03:32.378120283 +0000 UTC m=+161.238042745" observedRunningTime="2026-05-26 16:03:33.544508285 +0000 UTC m=+162.404430767" watchObservedRunningTime="2026-05-26 16:03:33.545240777 +0000 UTC m=+162.405163260" May 26 16:03:33.560009 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.559968 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-kqb24" podStartSLOduration=9.312610715 podStartE2EDuration="11.559953441s" podCreationTimestamp="2026-05-26 16:03:22 +0000 UTC" firstStartedPulling="2026-05-26 16:03:30.525375864 +0000 UTC m=+159.385298325" lastFinishedPulling="2026-05-26 16:03:32.772718596 +0000 UTC m=+161.632641051" observedRunningTime="2026-05-26 16:03:33.559375265 +0000 UTC m=+162.419297738" watchObservedRunningTime="2026-05-26 16:03:33.559953441 +0000 UTC m=+162.419875911" May 26 16:03:33.666371 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.666343 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l7p5g\"" May 26 16:03:33.666540 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.666491 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w25cq\"" May 26 16:03:33.673535 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.673510 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rs5nq" May 26 16:03:33.673677 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:33.673552 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8lq6" May 26 16:03:34.016595 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.016489 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p8lq6"] May 26 16:03:34.019771 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:34.019735 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42e8c71_312c_41a0_a8d9_b116879302ae.slice/crio-72e1248a4aea03eadcaf1712a67d5fad3ca462c74423d7526a725fbef046821a WatchSource:0}: Error finding container 72e1248a4aea03eadcaf1712a67d5fad3ca462c74423d7526a725fbef046821a: Status 404 returned error can't find the container with id 72e1248a4aea03eadcaf1712a67d5fad3ca462c74423d7526a725fbef046821a May 26 16:03:34.033673 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.033651 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rs5nq"] May 26 16:03:34.036003 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:03:34.035973 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3ca0a4_5dbd_425b_a0d3_3f2688605a3a.slice/crio-3ef04cf030caee04dd605a99446574a686a0ce91bbb3621026ebb2d40079d14c WatchSource:0}: Error finding container 3ef04cf030caee04dd605a99446574a686a0ce91bbb3621026ebb2d40079d14c: Status 404 returned error can't find the container with id 3ef04cf030caee04dd605a99446574a686a0ce91bbb3621026ebb2d40079d14c May 26 16:03:34.326677 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.326612 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-95767f565-hd7l2" podUID="8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" containerName="registry" containerID="cri-o://618c8d57115b4a32445b3cf107c8bf77def020514966f84aeb4803aa23270dde" gracePeriod=30 May 26 16:03:34.541298 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.541199 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" event={"ID":"0aa5eec1-ca38-4545-a077-e6c873b4c7c5","Type":"ContainerStarted","Data":"9a889637c0ea0c12bf774f79de172d7fc782485a759c4b3f8edb1c3c879aaf19"} May 26 16:03:34.544199 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.544169 2573 generic.go:358] "Generic (PLEG): container finished" podID="8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" containerID="618c8d57115b4a32445b3cf107c8bf77def020514966f84aeb4803aa23270dde" exitCode=0 May 26 16:03:34.544311 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.544255 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-95767f565-hd7l2" event={"ID":"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1","Type":"ContainerDied","Data":"618c8d57115b4a32445b3cf107c8bf77def020514966f84aeb4803aa23270dde"} May 26 16:03:34.546102 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.546049 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rs5nq" event={"ID":"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a","Type":"ContainerStarted","Data":"3ef04cf030caee04dd605a99446574a686a0ce91bbb3621026ebb2d40079d14c"} May 26 16:03:34.548935 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.548682 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8lq6" event={"ID":"c42e8c71-312c-41a0-a8d9-b116879302ae","Type":"ContainerStarted","Data":"72e1248a4aea03eadcaf1712a67d5fad3ca462c74423d7526a725fbef046821a"} May 26 16:03:34.620893 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.620863 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:03:34.698812 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.698774 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kzn6\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-kube-api-access-5kzn6\") pod \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " May 26 16:03:34.699006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.698858 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-image-registry-private-configuration\") pod \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " May 26 16:03:34.699006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.698911 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-certificates\") pod \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " May 26 16:03:34.699006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.698939 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-installation-pull-secrets\") pod \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " May 26 16:03:34.699006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.699004 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-trusted-ca\") pod \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " May 26 16:03:34.699228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.699061 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-ca-trust-extracted\") pod \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " May 26 16:03:34.699228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.699093 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-bound-sa-token\") pod \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " May 26 16:03:34.699228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.699142 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") pod \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\" (UID: \"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1\") " May 26 16:03:34.702594 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.702235 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:03:34.702594 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.702563 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:03:34.708186 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.708137 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:03:34.710159 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.710119 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:03:34.710992 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.710951 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-kube-api-access-5kzn6" (OuterVolumeSpecName: "kube-api-access-5kzn6") pod "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1"). InnerVolumeSpecName "kube-api-access-5kzn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:03:34.711194 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.711155 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:03:34.712787 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.712763 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:03:34.713512 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.713490 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" (UID: "8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:03:34.800701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.800492 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kzn6\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-kube-api-access-5kzn6\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:03:34.800701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.800530 2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-image-registry-private-configuration\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:03:34.800701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.800547 2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-certificates\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:03:34.800701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.800562 2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-installation-pull-secrets\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:03:34.800701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.800577 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-trusted-ca\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:03:34.800701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.800592 2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-ca-trust-extracted\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:03:34.800701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.800606 2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-bound-sa-token\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:03:34.800701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:34.800620 2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1-registry-tls\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:03:35.553587 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.553535 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-95767f565-hd7l2" event={"ID":"8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1","Type":"ContainerDied","Data":"97d715645a2b648c34dfcd9f7de4540cc37abc055e396197c511f0c5427f0149"} May 26 16:03:35.553587 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.553588 2573 scope.go:117] "RemoveContainer" containerID="618c8d57115b4a32445b3cf107c8bf77def020514966f84aeb4803aa23270dde" May 26 16:03:35.554564 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.553557 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-95767f565-hd7l2" May 26 16:03:35.580625 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.580600 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-95767f565-hd7l2"] May 26 16:03:35.586660 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.586580 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-95767f565-hd7l2"] May 26 16:03:35.844577 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.844476 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:35.844863 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.844840 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:35.846185 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.846154 2573 patch_prober.go:28] interesting pod/console-7dcfd8bd6f-ffgl7 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.24:8443/health\": dial tcp 10.133.0.24:8443: connect: connection refused" start-of-body= May 26 16:03:35.846291 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.846209 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7dcfd8bd6f-ffgl7" podUID="22adef2c-3729-488a-a440-b1bd060dd355" containerName="console" probeResult="failure" output="Get \"https://10.133.0.24:8443/health\": dial tcp 10.133.0.24:8443: connect: connection refused" May 26 16:03:35.875672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:35.875639 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" path="/var/lib/kubelet/pods/8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1/volumes" May 26 16:03:38.567294 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.567249 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" event={"ID":"0aa5eec1-ca38-4545-a077-e6c873b4c7c5","Type":"ContainerStarted","Data":"aa76e416ab8bcfe57eee788eaa7a3b7f0297860790f90e6b4a16cab0b2d2a247"} May 26 16:03:38.567294 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.567296 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" event={"ID":"0aa5eec1-ca38-4545-a077-e6c873b4c7c5","Type":"ContainerStarted","Data":"f940685514f1279a23f1d4a3394f2ba892bfb49221ebbd2760fc7b5e4e4df107"} May 26 16:03:38.568889 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.568863 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rs5nq" event={"ID":"ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a","Type":"ContainerStarted","Data":"9f39f9b17f9c3a3af9bfc2cb85f427a08fb9bed093778e986be0b164840415af"} May 26 16:03:38.570614 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.570586 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8lq6" event={"ID":"c42e8c71-312c-41a0-a8d9-b116879302ae","Type":"ContainerStarted","Data":"3edb2101ff30b9fca835997614a89fc107b65dd281e7897fb582fb99256eb612"} May 26 16:03:38.570727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.570620 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8lq6" event={"ID":"c42e8c71-312c-41a0-a8d9-b116879302ae","Type":"ContainerStarted","Data":"6ea4d511cd9ac43cdb3289c76542152d92a39c5520a76bf6af9ac25dcb64d6f9"} May 26 16:03:38.570727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.570715 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p8lq6" May 26 16:03:38.594010 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.593951 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8548648857-6pkmp" podStartSLOduration=8.717597524 podStartE2EDuration="15.593935386s" podCreationTimestamp="2026-05-26 16:03:23 +0000 UTC" firstStartedPulling="2026-05-26 16:03:30.563808371 +0000 UTC m=+159.423730820" lastFinishedPulling="2026-05-26 16:03:37.440146221 +0000 UTC m=+166.300068682" observedRunningTime="2026-05-26 16:03:38.59129782 +0000 UTC m=+167.451220423" watchObservedRunningTime="2026-05-26 16:03:38.593935386 +0000 UTC m=+167.453857857" May 26 16:03:38.620263 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.616711 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rs5nq" podStartSLOduration=130.209347571 podStartE2EDuration="2m13.616693687s" podCreationTimestamp="2026-05-26 16:01:25 +0000 UTC" firstStartedPulling="2026-05-26 16:03:34.038034363 +0000 UTC m=+162.897956812" lastFinishedPulling="2026-05-26 16:03:37.445380477 +0000 UTC m=+166.305302928" observedRunningTime="2026-05-26 16:03:38.613866963 +0000 UTC m=+167.473789429" watchObservedRunningTime="2026-05-26 16:03:38.616693687 +0000 UTC m=+167.476616157" May 26 16:03:38.659622 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.659562 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-p8lq6" podStartSLOduration=130.241138709 podStartE2EDuration="2m13.659539414s" podCreationTimestamp="2026-05-26 16:01:25 +0000 UTC" firstStartedPulling="2026-05-26 16:03:34.021750637 +0000 UTC m=+162.881673086" lastFinishedPulling="2026-05-26 16:03:37.44015133 +0000 UTC m=+166.300073791" observedRunningTime="2026-05-26 16:03:38.658094411 +0000 UTC m=+167.518016886" watchObservedRunningTime="2026-05-26 16:03:38.659539414 +0000 UTC m=+167.519461935" May 26 16:03:38.733621 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:38.731299 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cbfc9dc65-fzqqq"] May 26 16:03:39.554649 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:39.554622 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c47c44689-7g8p2" May 26 16:03:40.411554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:40.411519 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:03:45.849349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:45.849322 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:45.853285 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:45.853261 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:03:48.575855 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:48.575806 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p8lq6" May 26 16:03:53.618511 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:53.618476 2573 generic.go:358] "Generic (PLEG): container finished" podID="5ade83d5-dc58-49a9-b12d-0ef025b0d373" containerID="9cd9eb404468565cf08cd8a1ca33e840b784ecf9f397bc9329121ed156f123b9" exitCode=0 May 26 16:03:53.619026 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:53.618555 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" event={"ID":"5ade83d5-dc58-49a9-b12d-0ef025b0d373","Type":"ContainerDied","Data":"9cd9eb404468565cf08cd8a1ca33e840b784ecf9f397bc9329121ed156f123b9"} May 26 16:03:53.619026 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:53.619004 2573 scope.go:117] "RemoveContainer" containerID="9cd9eb404468565cf08cd8a1ca33e840b784ecf9f397bc9329121ed156f123b9" May 26 16:03:54.623569 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:54.623537 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-x8qn9" event={"ID":"5ade83d5-dc58-49a9-b12d-0ef025b0d373","Type":"ContainerStarted","Data":"63960f1eef084e077907f3f5ac747a47fee8ab08a049f10e14d51508b797440f"} May 26 16:03:59.160193 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:03:59.160157 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dcfd8bd6f-ffgl7"] May 26 16:04:03.754839 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:03.754767 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5cbfc9dc65-fzqqq" podUID="bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" containerName="console" containerID="cri-o://9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf" gracePeriod=15 May 26 16:04:04.003205 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.003183 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cbfc9dc65-fzqqq_bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0/console/0.log" May 26 16:04:04.003313 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.003243 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:04:04.038265 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.038238 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-oauth-serving-cert\") pod \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " May 26 16:04:04.038437 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.038299 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-trusted-ca-bundle\") pod \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " May 26 16:04:04.038437 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.038320 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnhj9\" (UniqueName: \"kubernetes.io/projected/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-kube-api-access-jnhj9\") pod \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " May 26 16:04:04.038437 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.038360 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-service-ca\") pod \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " May 26 16:04:04.038437 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.038389 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-config\") pod \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " May 26 16:04:04.038437 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.038422 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-serving-cert\") pod \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " May 26 16:04:04.038999 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.038971 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" (UID: "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:04.039166 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.039001 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" (UID: "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:04.039166 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.038997 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-config" (OuterVolumeSpecName: "console-config") pod "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" (UID: "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:04.039166 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.039016 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" (UID: "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:04.040788 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.040769 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" (UID: "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:04:04.040947 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.040927 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-kube-api-access-jnhj9" (OuterVolumeSpecName: "kube-api-access-jnhj9") pod "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" (UID: "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0"). InnerVolumeSpecName "kube-api-access-jnhj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:04:04.139588 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.139555 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-oauth-config\") pod \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\" (UID: \"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0\") " May 26 16:04:04.139748 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.139707 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-config\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:04.139748 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.139718 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-serving-cert\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:04.139748 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.139728 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-oauth-serving-cert\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:04.139748 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.139737 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-trusted-ca-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:04.139748 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.139746 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnhj9\" (UniqueName: \"kubernetes.io/projected/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-kube-api-access-jnhj9\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:04.139973 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.139755 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-service-ca\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:04.141504 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.141481 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" (UID: "bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:04:04.240095 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.240063 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0-console-oauth-config\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:04.655213 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.655186 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cbfc9dc65-fzqqq_bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0/console/0.log" May 26 16:04:04.655416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.655224 2573 generic.go:358] "Generic (PLEG): container finished" podID="bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" containerID="9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf" exitCode=2 May 26 16:04:04.655416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.655298 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbfc9dc65-fzqqq" May 26 16:04:04.655416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.655319 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbfc9dc65-fzqqq" event={"ID":"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0","Type":"ContainerDied","Data":"9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf"} May 26 16:04:04.655416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.655369 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbfc9dc65-fzqqq" event={"ID":"bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0","Type":"ContainerDied","Data":"771a0254fb558cdee2d47352978899ab75278085a89f2cc15627a18cf0a87f01"} May 26 16:04:04.655416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.655391 2573 scope.go:117] "RemoveContainer" containerID="9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf" May 26 16:04:04.664228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.664211 2573 scope.go:117] "RemoveContainer" containerID="9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf" May 26 16:04:04.664494 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:04:04.664472 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf\": container with ID starting with 9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf not found: ID does not exist" containerID="9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf" May 26 16:04:04.664554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.664502 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf"} err="failed to get container status \"9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf\": rpc error: code = NotFound desc = could not find container \"9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf\": container with ID starting with 9f5b41f1cd121e77ca6931ea1a280ed7cb7548c1f081b05c48b31633ed6fcecf not found: ID does not exist" May 26 16:04:04.681152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.681120 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cbfc9dc65-fzqqq"] May 26 16:04:04.685473 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:04.685450 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cbfc9dc65-fzqqq"] May 26 16:04:05.873022 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:05.872989 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" path="/var/lib/kubelet/pods/bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0/volumes" May 26 16:04:17.694519 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:17.694486 2573 generic.go:358] "Generic (PLEG): container finished" podID="317da9eb-b20e-413a-af49-fde829ee35d5" containerID="d9d48814584b950c8f07004f98afce0518e9eb2864fe2a9456d108d93bbd919b" exitCode=0 May 26 16:04:17.694914 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:17.694553 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" event={"ID":"317da9eb-b20e-413a-af49-fde829ee35d5","Type":"ContainerDied","Data":"d9d48814584b950c8f07004f98afce0518e9eb2864fe2a9456d108d93bbd919b"} May 26 16:04:17.694959 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:17.694934 2573 scope.go:117] "RemoveContainer" containerID="d9d48814584b950c8f07004f98afce0518e9eb2864fe2a9456d108d93bbd919b" May 26 16:04:18.424128 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:18.424100 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b4gnl_5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a/node-ca/0.log" May 26 16:04:18.699041 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:18.698957 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-tbzzs" event={"ID":"317da9eb-b20e-413a-af49-fde829ee35d5","Type":"ContainerStarted","Data":"87355c5aec1ea14c8e98aca5a963831de48f8c4130c5040913c81616044b4578"} May 26 16:04:24.179772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.179712 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7dcfd8bd6f-ffgl7" podUID="22adef2c-3729-488a-a440-b1bd060dd355" containerName="console" containerID="cri-o://3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b" gracePeriod=15 May 26 16:04:24.424550 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.424528 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7dcfd8bd6f-ffgl7_22adef2c-3729-488a-a440-b1bd060dd355/console/0.log" May 26 16:04:24.424656 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.424593 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:04:24.505261 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505186 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-oauth-serving-cert\") pod \"22adef2c-3729-488a-a440-b1bd060dd355\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " May 26 16:04:24.505261 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505225 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-oauth-config\") pod \"22adef2c-3729-488a-a440-b1bd060dd355\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " May 26 16:04:24.505261 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505252 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-serving-cert\") pod \"22adef2c-3729-488a-a440-b1bd060dd355\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " May 26 16:04:24.505518 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505288 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-console-config\") pod \"22adef2c-3729-488a-a440-b1bd060dd355\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " May 26 16:04:24.505518 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505311 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5lwb\" (UniqueName: \"kubernetes.io/projected/22adef2c-3729-488a-a440-b1bd060dd355-kube-api-access-p5lwb\") pod \"22adef2c-3729-488a-a440-b1bd060dd355\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " May 26 16:04:24.505518 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505353 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-trusted-ca-bundle\") pod \"22adef2c-3729-488a-a440-b1bd060dd355\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " May 26 16:04:24.505518 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505387 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-service-ca\") pod \"22adef2c-3729-488a-a440-b1bd060dd355\" (UID: \"22adef2c-3729-488a-a440-b1bd060dd355\") " May 26 16:04:24.505778 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505743 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "22adef2c-3729-488a-a440-b1bd060dd355" (UID: "22adef2c-3729-488a-a440-b1bd060dd355"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:24.505927 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505873 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-console-config" (OuterVolumeSpecName: "console-config") pod "22adef2c-3729-488a-a440-b1bd060dd355" (UID: "22adef2c-3729-488a-a440-b1bd060dd355"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:24.505927 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505898 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-service-ca" (OuterVolumeSpecName: "service-ca") pod "22adef2c-3729-488a-a440-b1bd060dd355" (UID: "22adef2c-3729-488a-a440-b1bd060dd355"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:24.506023 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.505936 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "22adef2c-3729-488a-a440-b1bd060dd355" (UID: "22adef2c-3729-488a-a440-b1bd060dd355"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:04:24.507427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.507406 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "22adef2c-3729-488a-a440-b1bd060dd355" (UID: "22adef2c-3729-488a-a440-b1bd060dd355"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:04:24.507516 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.507458 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "22adef2c-3729-488a-a440-b1bd060dd355" (UID: "22adef2c-3729-488a-a440-b1bd060dd355"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:04:24.507516 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.507470 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22adef2c-3729-488a-a440-b1bd060dd355-kube-api-access-p5lwb" (OuterVolumeSpecName: "kube-api-access-p5lwb") pod "22adef2c-3729-488a-a440-b1bd060dd355" (UID: "22adef2c-3729-488a-a440-b1bd060dd355"). InnerVolumeSpecName "kube-api-access-p5lwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:04:24.606627 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.606585 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-oauth-serving-cert\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:24.606627 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.606619 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-oauth-config\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:24.606627 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.606631 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22adef2c-3729-488a-a440-b1bd060dd355-console-serving-cert\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:24.606902 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.606646 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-console-config\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:24.606902 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.606658 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5lwb\" (UniqueName: \"kubernetes.io/projected/22adef2c-3729-488a-a440-b1bd060dd355-kube-api-access-p5lwb\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:24.606902 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.606671 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-trusted-ca-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:24.606902 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.606682 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22adef2c-3729-488a-a440-b1bd060dd355-service-ca\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:04:24.720919 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.720890 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7dcfd8bd6f-ffgl7_22adef2c-3729-488a-a440-b1bd060dd355/console/0.log" May 26 16:04:24.721084 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.720927 2573 generic.go:358] "Generic (PLEG): container finished" podID="22adef2c-3729-488a-a440-b1bd060dd355" containerID="3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b" exitCode=2 May 26 16:04:24.721084 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.721000 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcfd8bd6f-ffgl7" May 26 16:04:24.721084 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.721016 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcfd8bd6f-ffgl7" event={"ID":"22adef2c-3729-488a-a440-b1bd060dd355","Type":"ContainerDied","Data":"3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b"} May 26 16:04:24.721084 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.721058 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcfd8bd6f-ffgl7" event={"ID":"22adef2c-3729-488a-a440-b1bd060dd355","Type":"ContainerDied","Data":"b6dd228a2888a6ee34df1330d20cba6f5ebb2f8ea848f24f9cb217dff09906e9"} May 26 16:04:24.721084 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.721074 2573 scope.go:117] "RemoveContainer" containerID="3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b" May 26 16:04:24.729167 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:04:24.729139 2573 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22adef2c_3729_488a_a440_b1bd060dd355.slice\": RecentStats: unable to find data in memory cache]" May 26 16:04:24.729625 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.729610 2573 scope.go:117] "RemoveContainer" containerID="3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b" May 26 16:04:24.729908 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:04:24.729886 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b\": container with ID starting with 3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b not found: ID does not exist" containerID="3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b" May 26 16:04:24.729991 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.729913 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b"} err="failed to get container status \"3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b\": rpc error: code = NotFound desc = could not find container \"3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b\": container with ID starting with 3a6d506c7554ee2739c671db93ca5fab3ab1088c60553c5bd55b9fd53e901f8b not found: ID does not exist" May 26 16:04:24.743603 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.743573 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dcfd8bd6f-ffgl7"] May 26 16:04:24.748144 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:24.748121 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dcfd8bd6f-ffgl7"] May 26 16:04:25.873585 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:25.873542 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22adef2c-3729-488a-a440-b1bd060dd355" path="/var/lib/kubelet/pods/22adef2c-3729-488a-a440-b1bd060dd355/volumes" May 26 16:04:36.915724 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.915679 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c6c47dcd7-qbzfk"] May 26 16:04:36.916197 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.916143 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" containerName="registry" May 26 16:04:36.916197 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.916164 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" containerName="registry" May 26 16:04:36.916197 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.916176 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" containerName="console" May 26 16:04:36.916197 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.916184 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" containerName="console" May 26 16:04:36.916421 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.916202 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22adef2c-3729-488a-a440-b1bd060dd355" containerName="console" May 26 16:04:36.916421 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.916211 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="22adef2c-3729-488a-a440-b1bd060dd355" containerName="console" May 26 16:04:36.916421 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.916300 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="8ecffbe8-3f45-4d4f-aaaa-c67430c55ac1" containerName="registry" May 26 16:04:36.916421 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.916317 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc3cc2ff-246f-48cf-88ce-338b1eb2c4a0" containerName="console" May 26 16:04:36.916421 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.916326 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="22adef2c-3729-488a-a440-b1bd060dd355" containerName="console" May 26 16:04:36.919604 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.919583 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:36.922178 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.922155 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" May 26 16:04:36.922284 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.922156 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ssfl9\"" May 26 16:04:36.922809 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.922795 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" May 26 16:04:36.923624 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.923607 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" May 26 16:04:36.923696 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.923657 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" May 26 16:04:36.924496 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.924477 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" May 26 16:04:36.934279 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.934256 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" May 26 16:04:36.937851 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:36.937811 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c6c47dcd7-qbzfk"] May 26 16:04:37.016343 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.016308 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-oauth-config\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.016513 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.016362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-serving-cert\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.016513 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.016380 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-service-ca\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.016513 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.016407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99262\" (UniqueName: \"kubernetes.io/projected/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-kube-api-access-99262\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.016513 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.016448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-oauth-serving-cert\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.016513 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.016471 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-trusted-ca-bundle\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.016513 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.016495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-config\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.117656 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.117605 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-oauth-config\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.117856 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.117775 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-serving-cert\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.117856 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.117818 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-service-ca\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.117949 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.117877 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99262\" (UniqueName: \"kubernetes.io/projected/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-kube-api-access-99262\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.117949 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.117910 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-oauth-serving-cert\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.118025 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.117949 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-trusted-ca-bundle\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.118116 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.118092 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-config\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.118619 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.118592 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-service-ca\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.118715 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.118592 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-oauth-serving-cert\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.118715 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.118662 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-config\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.118861 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.118813 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-trusted-ca-bundle\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.120324 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.120302 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-oauth-config\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.120398 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.120384 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-serving-cert\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.125721 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.125700 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99262\" (UniqueName: \"kubernetes.io/projected/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-kube-api-access-99262\") pod \"console-5c6c47dcd7-qbzfk\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.229451 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.229356 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:37.358045 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.358022 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c6c47dcd7-qbzfk"] May 26 16:04:37.359918 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:04:37.359890 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb901b9_c7d8_49cf_99e5_04e1f19c8031.slice/crio-2d8f2bc4212745e6d6187cf65a4e5fb34f82fd3e8a040198e04d05b0f38d1aca WatchSource:0}: Error finding container 2d8f2bc4212745e6d6187cf65a4e5fb34f82fd3e8a040198e04d05b0f38d1aca: Status 404 returned error can't find the container with id 2d8f2bc4212745e6d6187cf65a4e5fb34f82fd3e8a040198e04d05b0f38d1aca May 26 16:04:37.762082 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.762049 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6c47dcd7-qbzfk" event={"ID":"6fb901b9-c7d8-49cf-99e5-04e1f19c8031","Type":"ContainerStarted","Data":"7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9"} May 26 16:04:37.762082 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.762084 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6c47dcd7-qbzfk" event={"ID":"6fb901b9-c7d8-49cf-99e5-04e1f19c8031","Type":"ContainerStarted","Data":"2d8f2bc4212745e6d6187cf65a4e5fb34f82fd3e8a040198e04d05b0f38d1aca"} May 26 16:04:37.782379 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:37.782327 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5c6c47dcd7-qbzfk" podStartSLOduration=1.782313118 podStartE2EDuration="1.782313118s" podCreationTimestamp="2026-05-26 16:04:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:04:37.781986425 +0000 UTC m=+226.641908893" watchObservedRunningTime="2026-05-26 16:04:37.782313118 +0000 UTC m=+226.642235587" May 26 16:04:39.807999 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.807969 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 26 16:04:39.810699 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.810684 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.813131 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.813090 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" May 26 16:04:39.813276 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.813090 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" May 26 16:04:39.813276 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.813259 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" May 26 16:04:39.813488 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.813471 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gdncm\"" May 26 16:04:39.813524 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.813501 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" May 26 16:04:39.813607 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.813573 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" May 26 16:04:39.813607 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.813580 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" May 26 16:04:39.813798 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.813625 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" May 26 16:04:39.813798 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.813606 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" May 26 16:04:39.819442 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.819425 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" May 26 16:04:39.823733 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.823711 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 26 16:04:39.943812 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.943775 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.943812 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.943818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-web-config\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944026 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.943860 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bb6bbc86-422e-4657-8ef7-0351f07efa06-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944026 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.943916 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-config-volume\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944026 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.943966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944121 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.944024 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb6bbc86-422e-4657-8ef7-0351f07efa06-config-out\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944121 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.944058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944121 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.944090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb6bbc86-422e-4657-8ef7-0351f07efa06-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944121 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.944112 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944236 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.944129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb6bbc86-422e-4657-8ef7-0351f07efa06-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944236 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.944147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944236 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.944196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbn4d\" (UniqueName: \"kubernetes.io/projected/bb6bbc86-422e-4657-8ef7-0351f07efa06-kube-api-access-kbn4d\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:39.944236 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:39.944219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb6bbc86-422e-4657-8ef7-0351f07efa06-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045108 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bb6bbc86-422e-4657-8ef7-0351f07efa06-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045155 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-config-volume\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045187 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045207 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb6bbc86-422e-4657-8ef7-0351f07efa06-config-out\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045231 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045252 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb6bbc86-422e-4657-8ef7-0351f07efa06-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045278 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045304 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb6bbc86-422e-4657-8ef7-0351f07efa06-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045327 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045366 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kbn4d\" (UniqueName: \"kubernetes.io/projected/bb6bbc86-422e-4657-8ef7-0351f07efa06-kube-api-access-kbn4d\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045780 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045403 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb6bbc86-422e-4657-8ef7-0351f07efa06-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045780 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045434 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045780 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045467 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-web-config\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.045780 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.045579 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bb6bbc86-422e-4657-8ef7-0351f07efa06-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.046254 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.046225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb6bbc86-422e-4657-8ef7-0351f07efa06-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.048346 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.048245 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb6bbc86-422e-4657-8ef7-0351f07efa06-config-out\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.048632 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.048523 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.048632 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.048523 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-config-volume\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.048797 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.048653 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.048797 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.048780 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.048941 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.048858 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb6bbc86-422e-4657-8ef7-0351f07efa06-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.049034 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.049006 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb6bbc86-422e-4657-8ef7-0351f07efa06-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.049191 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.049174 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.049443 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.049425 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-web-config\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.050499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.050480 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bb6bbc86-422e-4657-8ef7-0351f07efa06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.058236 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.058192 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbn4d\" (UniqueName: \"kubernetes.io/projected/bb6bbc86-422e-4657-8ef7-0351f07efa06-kube-api-access-kbn4d\") pod \"alertmanager-main-0\" (UID: \"bb6bbc86-422e-4657-8ef7-0351f07efa06\") " pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.121548 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.121518 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 26 16:04:40.257923 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.257856 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 26 16:04:40.260631 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:04:40.260600 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6bbc86_422e_4657_8ef7_0351f07efa06.slice/crio-9dee7eff0cb90346a560e45cb863f63e52bcdd68268bb69c30c523a89db8653d WatchSource:0}: Error finding container 9dee7eff0cb90346a560e45cb863f63e52bcdd68268bb69c30c523a89db8653d: Status 404 returned error can't find the container with id 9dee7eff0cb90346a560e45cb863f63e52bcdd68268bb69c30c523a89db8653d May 26 16:04:40.772285 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.772251 2573 generic.go:358] "Generic (PLEG): container finished" podID="bb6bbc86-422e-4657-8ef7-0351f07efa06" containerID="2eeab81d84da0b256f04a11dbcb2e9b882943ea9778ef72498865ba8e5766061" exitCode=0 May 26 16:04:40.772450 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.772339 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bb6bbc86-422e-4657-8ef7-0351f07efa06","Type":"ContainerDied","Data":"2eeab81d84da0b256f04a11dbcb2e9b882943ea9778ef72498865ba8e5766061"} May 26 16:04:40.772450 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:40.772375 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bb6bbc86-422e-4657-8ef7-0351f07efa06","Type":"ContainerStarted","Data":"9dee7eff0cb90346a560e45cb863f63e52bcdd68268bb69c30c523a89db8653d"} May 26 16:04:42.784050 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:42.784013 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bb6bbc86-422e-4657-8ef7-0351f07efa06","Type":"ContainerStarted","Data":"c36b395242069980d40148cc5633fca61b1647eeb6a0875fbc210004443ccd6c"} May 26 16:04:42.784050 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:42.784055 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bb6bbc86-422e-4657-8ef7-0351f07efa06","Type":"ContainerStarted","Data":"6d2877f12296ae7b42449112ff66943714963cd290e6b60110198e8778539224"} May 26 16:04:42.784471 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:42.784065 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bb6bbc86-422e-4657-8ef7-0351f07efa06","Type":"ContainerStarted","Data":"51f251c6d07216cdf3b879907c6f912d09edfd2d113d3c9c1f89b24eb5af70e0"} May 26 16:04:42.784471 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:42.784074 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bb6bbc86-422e-4657-8ef7-0351f07efa06","Type":"ContainerStarted","Data":"5cd745e223cde6c2ebca9180d33b2951041672b23d95fd3f55e5c4bf16f8536e"} May 26 16:04:42.784471 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:42.784082 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bb6bbc86-422e-4657-8ef7-0351f07efa06","Type":"ContainerStarted","Data":"05860a1d452870c6317bb685d04bad9bac9ea7a46a2b1b8e0daf86161770da6f"} May 26 16:04:42.784471 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:42.784090 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bb6bbc86-422e-4657-8ef7-0351f07efa06","Type":"ContainerStarted","Data":"11b10bdadc5f6b89e852ae06f487deae10f8f2b7928a1ef135c4fb8ba7feeded"} May 26 16:04:42.812658 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:42.812605 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.709376713 podStartE2EDuration="3.812588637s" podCreationTimestamp="2026-05-26 16:04:39 +0000 UTC" firstStartedPulling="2026-05-26 16:04:40.773453057 +0000 UTC m=+229.633375505" lastFinishedPulling="2026-05-26 16:04:41.876664981 +0000 UTC m=+230.736587429" observedRunningTime="2026-05-26 16:04:42.811264294 +0000 UTC m=+231.671186763" watchObservedRunningTime="2026-05-26 16:04:42.812588637 +0000 UTC m=+231.672511108" May 26 16:04:47.229706 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:47.229671 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:47.229706 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:47.229706 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:47.234354 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:47.234332 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:04:47.805985 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:04:47.805954 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:05:51.740345 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:05:51.740316 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:05:51.741913 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:05:51.741892 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:05:51.750222 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:05:51.750197 2573 kubelet.go:1628] "Image garbage collection succeeded" May 26 16:07:11.804413 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.801321 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98"] May 26 16:07:11.806475 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.806448 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:11.809298 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.809279 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" May 26 16:07:11.809976 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.809959 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" May 26 16:07:11.810081 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.809997 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dt9sc\"" May 26 16:07:11.814522 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.814504 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98"] May 26 16:07:11.893586 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.893555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:11.893586 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.893592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:11.893788 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.893624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt2kn\" (UniqueName: \"kubernetes.io/projected/3c515017-0c03-40dc-8956-027da5e34032-kube-api-access-zt2kn\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:11.994761 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.994722 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2kn\" (UniqueName: \"kubernetes.io/projected/3c515017-0c03-40dc-8956-027da5e34032-kube-api-access-zt2kn\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:11.994948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.994856 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:11.995021 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.994999 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:11.995271 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.995254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:11.995325 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:11.995309 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:12.004813 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:12.004778 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2kn\" (UniqueName: \"kubernetes.io/projected/3c515017-0c03-40dc-8956-027da5e34032-kube-api-access-zt2kn\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:12.116604 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:12.116518 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:12.245354 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:12.245330 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98"] May 26 16:07:12.247735 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:07:12.247706 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c515017_0c03_40dc_8956_027da5e34032.slice/crio-1b83325b3ec5e82cf3d5163ba6604dad3ab1f0400616aa43967516a473597abe WatchSource:0}: Error finding container 1b83325b3ec5e82cf3d5163ba6604dad3ab1f0400616aa43967516a473597abe: Status 404 returned error can't find the container with id 1b83325b3ec5e82cf3d5163ba6604dad3ab1f0400616aa43967516a473597abe May 26 16:07:12.249445 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:12.249428 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:07:13.218894 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:13.218853 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" event={"ID":"3c515017-0c03-40dc-8956-027da5e34032","Type":"ContainerStarted","Data":"1b83325b3ec5e82cf3d5163ba6604dad3ab1f0400616aa43967516a473597abe"} May 26 16:07:21.245872 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:21.245774 2573 generic.go:358] "Generic (PLEG): container finished" podID="3c515017-0c03-40dc-8956-027da5e34032" containerID="826ba31d1640ce851ed9732bf9367110f416a18184c64eae40b8d89eee53a9fd" exitCode=0 May 26 16:07:21.245872 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:21.245861 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" event={"ID":"3c515017-0c03-40dc-8956-027da5e34032","Type":"ContainerDied","Data":"826ba31d1640ce851ed9732bf9367110f416a18184c64eae40b8d89eee53a9fd"} May 26 16:07:23.254916 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:23.254871 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" event={"ID":"3c515017-0c03-40dc-8956-027da5e34032","Type":"ContainerStarted","Data":"0ae1c9d24c768f71aabbf2153fe8543aacc8691f628e3cbd3feb703f5c12f0fe"} May 26 16:07:24.259509 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:24.259470 2573 generic.go:358] "Generic (PLEG): container finished" podID="3c515017-0c03-40dc-8956-027da5e34032" containerID="0ae1c9d24c768f71aabbf2153fe8543aacc8691f628e3cbd3feb703f5c12f0fe" exitCode=0 May 26 16:07:24.259915 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:24.259551 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" event={"ID":"3c515017-0c03-40dc-8956-027da5e34032","Type":"ContainerDied","Data":"0ae1c9d24c768f71aabbf2153fe8543aacc8691f628e3cbd3feb703f5c12f0fe"} May 26 16:07:31.284795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:31.284758 2573 generic.go:358] "Generic (PLEG): container finished" podID="3c515017-0c03-40dc-8956-027da5e34032" containerID="58cfbaf5c9a353b3c21eba00b3586c3f0c096be0f90dd6fd3359cb3cadc2430f" exitCode=0 May 26 16:07:31.285206 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:31.284819 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" event={"ID":"3c515017-0c03-40dc-8956-027da5e34032","Type":"ContainerDied","Data":"58cfbaf5c9a353b3c21eba00b3586c3f0c096be0f90dd6fd3359cb3cadc2430f"} May 26 16:07:32.410948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.410926 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:32.473150 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.473118 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-bundle\") pod \"3c515017-0c03-40dc-8956-027da5e34032\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " May 26 16:07:32.473289 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.473176 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt2kn\" (UniqueName: \"kubernetes.io/projected/3c515017-0c03-40dc-8956-027da5e34032-kube-api-access-zt2kn\") pod \"3c515017-0c03-40dc-8956-027da5e34032\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " May 26 16:07:32.473289 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.473217 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-util\") pod \"3c515017-0c03-40dc-8956-027da5e34032\" (UID: \"3c515017-0c03-40dc-8956-027da5e34032\") " May 26 16:07:32.473871 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.473819 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-bundle" (OuterVolumeSpecName: "bundle") pod "3c515017-0c03-40dc-8956-027da5e34032" (UID: "3c515017-0c03-40dc-8956-027da5e34032"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:07:32.475365 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.475343 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c515017-0c03-40dc-8956-027da5e34032-kube-api-access-zt2kn" (OuterVolumeSpecName: "kube-api-access-zt2kn") pod "3c515017-0c03-40dc-8956-027da5e34032" (UID: "3c515017-0c03-40dc-8956-027da5e34032"). InnerVolumeSpecName "kube-api-access-zt2kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:07:32.477496 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.477477 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-util" (OuterVolumeSpecName: "util") pod "3c515017-0c03-40dc-8956-027da5e34032" (UID: "3c515017-0c03-40dc-8956-027da5e34032"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:07:32.574511 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.574437 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:07:32.574511 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.574468 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt2kn\" (UniqueName: \"kubernetes.io/projected/3c515017-0c03-40dc-8956-027da5e34032-kube-api-access-zt2kn\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:07:32.574511 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:32.574477 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c515017-0c03-40dc-8956-027da5e34032-util\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:07:33.291502 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:33.291470 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" event={"ID":"3c515017-0c03-40dc-8956-027da5e34032","Type":"ContainerDied","Data":"1b83325b3ec5e82cf3d5163ba6604dad3ab1f0400616aa43967516a473597abe"} May 26 16:07:33.291502 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:33.291505 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b83325b3ec5e82cf3d5163ba6604dad3ab1f0400616aa43967516a473597abe" May 26 16:07:33.291699 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:33.291508 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccjv98" May 26 16:07:38.658354 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.658315 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk"] May 26 16:07:38.658948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.658663 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c515017-0c03-40dc-8956-027da5e34032" containerName="pull" May 26 16:07:38.658948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.658675 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c515017-0c03-40dc-8956-027da5e34032" containerName="pull" May 26 16:07:38.658948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.658688 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c515017-0c03-40dc-8956-027da5e34032" containerName="extract" May 26 16:07:38.658948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.658693 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c515017-0c03-40dc-8956-027da5e34032" containerName="extract" May 26 16:07:38.658948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.658708 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c515017-0c03-40dc-8956-027da5e34032" containerName="util" May 26 16:07:38.658948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.658714 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c515017-0c03-40dc-8956-027da5e34032" containerName="util" May 26 16:07:38.658948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.658767 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c515017-0c03-40dc-8956-027da5e34032" containerName="extract" May 26 16:07:38.662558 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.662538 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:07:38.665141 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.665121 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" May 26 16:07:38.665279 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.665158 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-blbfh\"" May 26 16:07:38.665279 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.665234 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" May 26 16:07:38.665279 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.665246 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" May 26 16:07:38.674925 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.674906 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk"] May 26 16:07:38.727212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.727165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/fe23b32e-1232-44a2-ae46-9ea6a2b49f92-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk\" (UID: \"fe23b32e-1232-44a2-ae46-9ea6a2b49f92\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:07:38.727358 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.727230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9djsh\" (UniqueName: \"kubernetes.io/projected/fe23b32e-1232-44a2-ae46-9ea6a2b49f92-kube-api-access-9djsh\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk\" (UID: \"fe23b32e-1232-44a2-ae46-9ea6a2b49f92\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:07:38.828278 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.828238 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/fe23b32e-1232-44a2-ae46-9ea6a2b49f92-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk\" (UID: \"fe23b32e-1232-44a2-ae46-9ea6a2b49f92\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:07:38.828466 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.828292 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9djsh\" (UniqueName: \"kubernetes.io/projected/fe23b32e-1232-44a2-ae46-9ea6a2b49f92-kube-api-access-9djsh\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk\" (UID: \"fe23b32e-1232-44a2-ae46-9ea6a2b49f92\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:07:38.830692 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.830668 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/fe23b32e-1232-44a2-ae46-9ea6a2b49f92-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk\" (UID: \"fe23b32e-1232-44a2-ae46-9ea6a2b49f92\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:07:38.841290 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.841267 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9djsh\" (UniqueName: \"kubernetes.io/projected/fe23b32e-1232-44a2-ae46-9ea6a2b49f92-kube-api-access-9djsh\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk\" (UID: \"fe23b32e-1232-44a2-ae46-9ea6a2b49f92\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:07:38.973020 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:38.972936 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:07:39.101996 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:39.101944 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk"] May 26 16:07:39.104583 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:07:39.104549 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe23b32e_1232_44a2_ae46_9ea6a2b49f92.slice/crio-0f9ff4e61fb9b9106f2548e1d583be9325cc639ff08fbb057156af1760b6857f WatchSource:0}: Error finding container 0f9ff4e61fb9b9106f2548e1d583be9325cc639ff08fbb057156af1760b6857f: Status 404 returned error can't find the container with id 0f9ff4e61fb9b9106f2548e1d583be9325cc639ff08fbb057156af1760b6857f May 26 16:07:39.309719 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:39.309683 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" event={"ID":"fe23b32e-1232-44a2-ae46-9ea6a2b49f92","Type":"ContainerStarted","Data":"0f9ff4e61fb9b9106f2548e1d583be9325cc639ff08fbb057156af1760b6857f"} May 26 16:07:44.834476 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.834440 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-7z7vw"] May 26 16:07:44.836663 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.836642 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:44.839232 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.839211 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" May 26 16:07:44.839357 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.839239 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hcztz\"" May 26 16:07:44.839357 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.839328 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" May 26 16:07:44.846291 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.846274 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-7z7vw"] May 26 16:07:44.879939 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.879918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:44.880045 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.879999 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npjc8\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-kube-api-access-npjc8\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:44.880045 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.880029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/0cd733a6-f301-4fc5-988e-ddc2626b4968-cabundle0\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:44.980868 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.980849 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npjc8\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-kube-api-access-npjc8\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:44.980974 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.980886 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/0cd733a6-f301-4fc5-988e-ddc2626b4968-cabundle0\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:44.980974 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.980956 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:44.981243 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:44.981223 2573 secret.go:281] references non-existent secret key: ca.crt May 26 16:07:44.981315 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:44.981248 2573 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt May 26 16:07:44.981315 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:44.981260 2573 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-7z7vw: references non-existent secret key: ca.crt May 26 16:07:44.981454 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:44.981322 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates podName:0cd733a6-f301-4fc5-988e-ddc2626b4968 nodeName:}" failed. No retries permitted until 2026-05-26 16:07:45.481304701 +0000 UTC m=+414.341227151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates") pod "keda-operator-ffbb595cb-7z7vw" (UID: "0cd733a6-f301-4fc5-988e-ddc2626b4968") : references non-existent secret key: ca.crt May 26 16:07:44.981695 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.981674 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/0cd733a6-f301-4fc5-988e-ddc2626b4968-cabundle0\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:44.989764 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:44.989745 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjc8\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-kube-api-access-npjc8\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:45.332268 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.332235 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" event={"ID":"fe23b32e-1232-44a2-ae46-9ea6a2b49f92","Type":"ContainerStarted","Data":"8bcfc2a62d323192fa54973023012e62599dd6b73a4e947de404f7889f02d609"} May 26 16:07:45.332438 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.332316 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:07:45.359977 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.359927 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" podStartSLOduration=2.175037011 podStartE2EDuration="7.359908828s" podCreationTimestamp="2026-05-26 16:07:38 +0000 UTC" firstStartedPulling="2026-05-26 16:07:39.106330202 +0000 UTC m=+407.966252649" lastFinishedPulling="2026-05-26 16:07:44.291202016 +0000 UTC m=+413.151124466" observedRunningTime="2026-05-26 16:07:45.358299949 +0000 UTC m=+414.218222417" watchObservedRunningTime="2026-05-26 16:07:45.359908828 +0000 UTC m=+414.219831303" May 26 16:07:45.416065 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.416038 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-9cpnv"] May 26 16:07:45.418210 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.418194 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:45.420470 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.420455 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" May 26 16:07:45.430357 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.430337 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-9cpnv"] May 26 16:07:45.486683 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.486656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/011be0c9-0c9c-4008-804b-fed4e30d51ae-certificates\") pod \"keda-admission-cf49989db-9cpnv\" (UID: \"011be0c9-0c9c-4008-804b-fed4e30d51ae\") " pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:45.486809 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.486785 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:45.486879 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.486846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pblwh\" (UniqueName: \"kubernetes.io/projected/011be0c9-0c9c-4008-804b-fed4e30d51ae-kube-api-access-pblwh\") pod \"keda-admission-cf49989db-9cpnv\" (UID: \"011be0c9-0c9c-4008-804b-fed4e30d51ae\") " pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:45.486930 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:45.486913 2573 secret.go:281] references non-existent secret key: ca.crt May 26 16:07:45.486992 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:45.486929 2573 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt May 26 16:07:45.486992 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:45.486940 2573 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-7z7vw: references non-existent secret key: ca.crt May 26 16:07:45.487073 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:45.487002 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates podName:0cd733a6-f301-4fc5-988e-ddc2626b4968 nodeName:}" failed. No retries permitted until 2026-05-26 16:07:46.486985396 +0000 UTC m=+415.346907861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates") pod "keda-operator-ffbb595cb-7z7vw" (UID: "0cd733a6-f301-4fc5-988e-ddc2626b4968") : references non-existent secret key: ca.crt May 26 16:07:45.587467 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.587375 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pblwh\" (UniqueName: \"kubernetes.io/projected/011be0c9-0c9c-4008-804b-fed4e30d51ae-kube-api-access-pblwh\") pod \"keda-admission-cf49989db-9cpnv\" (UID: \"011be0c9-0c9c-4008-804b-fed4e30d51ae\") " pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:45.587467 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.587444 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/011be0c9-0c9c-4008-804b-fed4e30d51ae-certificates\") pod \"keda-admission-cf49989db-9cpnv\" (UID: \"011be0c9-0c9c-4008-804b-fed4e30d51ae\") " pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:45.587649 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:45.587542 2573 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found May 26 16:07:45.587649 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:45.587570 2573 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-9cpnv: secret "keda-admission-webhooks-certs" not found May 26 16:07:45.587649 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:45.587619 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/011be0c9-0c9c-4008-804b-fed4e30d51ae-certificates podName:011be0c9-0c9c-4008-804b-fed4e30d51ae nodeName:}" failed. No retries permitted until 2026-05-26 16:07:46.087604878 +0000 UTC m=+414.947527325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/011be0c9-0c9c-4008-804b-fed4e30d51ae-certificates") pod "keda-admission-cf49989db-9cpnv" (UID: "011be0c9-0c9c-4008-804b-fed4e30d51ae") : secret "keda-admission-webhooks-certs" not found May 26 16:07:45.596227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:45.596196 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pblwh\" (UniqueName: \"kubernetes.io/projected/011be0c9-0c9c-4008-804b-fed4e30d51ae-kube-api-access-pblwh\") pod \"keda-admission-cf49989db-9cpnv\" (UID: \"011be0c9-0c9c-4008-804b-fed4e30d51ae\") " pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:46.091118 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:46.091078 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/011be0c9-0c9c-4008-804b-fed4e30d51ae-certificates\") pod \"keda-admission-cf49989db-9cpnv\" (UID: \"011be0c9-0c9c-4008-804b-fed4e30d51ae\") " pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:46.093576 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:46.093556 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/011be0c9-0c9c-4008-804b-fed4e30d51ae-certificates\") pod \"keda-admission-cf49989db-9cpnv\" (UID: \"011be0c9-0c9c-4008-804b-fed4e30d51ae\") " pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:46.328390 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:46.328360 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:46.455801 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:46.455776 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-9cpnv"] May 26 16:07:46.459053 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:07:46.459018 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011be0c9_0c9c_4008_804b_fed4e30d51ae.slice/crio-b119925b243e75deeb0e17237e446203f1780327daa1b4dafa72f6c7282788ef WatchSource:0}: Error finding container b119925b243e75deeb0e17237e446203f1780327daa1b4dafa72f6c7282788ef: Status 404 returned error can't find the container with id b119925b243e75deeb0e17237e446203f1780327daa1b4dafa72f6c7282788ef May 26 16:07:46.494944 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:46.494923 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:46.495049 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:46.495025 2573 secret.go:281] references non-existent secret key: ca.crt May 26 16:07:46.495049 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:46.495036 2573 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt May 26 16:07:46.495049 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:46.495044 2573 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-7z7vw: references non-existent secret key: ca.crt May 26 16:07:46.495148 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:46.495086 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates podName:0cd733a6-f301-4fc5-988e-ddc2626b4968 nodeName:}" failed. No retries permitted until 2026-05-26 16:07:48.495073063 +0000 UTC m=+417.354995509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates") pod "keda-operator-ffbb595cb-7z7vw" (UID: "0cd733a6-f301-4fc5-988e-ddc2626b4968") : references non-existent secret key: ca.crt May 26 16:07:47.340284 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:47.340247 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-9cpnv" event={"ID":"011be0c9-0c9c-4008-804b-fed4e30d51ae","Type":"ContainerStarted","Data":"b119925b243e75deeb0e17237e446203f1780327daa1b4dafa72f6c7282788ef"} May 26 16:07:48.511143 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:48.511103 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:48.511506 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:48.511261 2573 secret.go:281] references non-existent secret key: ca.crt May 26 16:07:48.511506 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:48.511281 2573 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt May 26 16:07:48.511506 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:48.511291 2573 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-7z7vw: references non-existent secret key: ca.crt May 26 16:07:48.511506 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:07:48.511343 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates podName:0cd733a6-f301-4fc5-988e-ddc2626b4968 nodeName:}" failed. No retries permitted until 2026-05-26 16:07:52.511328738 +0000 UTC m=+421.371251184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates") pod "keda-operator-ffbb595cb-7z7vw" (UID: "0cd733a6-f301-4fc5-988e-ddc2626b4968") : references non-existent secret key: ca.crt May 26 16:07:49.349452 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:49.349412 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-9cpnv" event={"ID":"011be0c9-0c9c-4008-804b-fed4e30d51ae","Type":"ContainerStarted","Data":"d72b57559d05d2e939635ffa186602c1ae0098d9a47e0beb3516cdf832f73c95"} May 26 16:07:49.349624 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:49.349514 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:07:49.366782 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:49.366736 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-9cpnv" podStartSLOduration=2.460288803 podStartE2EDuration="4.36672502s" podCreationTimestamp="2026-05-26 16:07:45 +0000 UTC" firstStartedPulling="2026-05-26 16:07:46.460671097 +0000 UTC m=+415.320593543" lastFinishedPulling="2026-05-26 16:07:48.367107299 +0000 UTC m=+417.227029760" observedRunningTime="2026-05-26 16:07:49.366106237 +0000 UTC m=+418.226028710" watchObservedRunningTime="2026-05-26 16:07:49.36672502 +0000 UTC m=+418.226647488" May 26 16:07:52.548329 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:52.548284 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:52.550898 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:52.550873 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0cd733a6-f301-4fc5-988e-ddc2626b4968-certificates\") pod \"keda-operator-ffbb595cb-7z7vw\" (UID: \"0cd733a6-f301-4fc5-988e-ddc2626b4968\") " pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:52.648351 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:52.648322 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:07:52.775578 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:52.775552 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-7z7vw"] May 26 16:07:52.777809 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:07:52.777782 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd733a6_f301_4fc5_988e_ddc2626b4968.slice/crio-dffee8bdfb7f2d7e557dbe805797aadba9dadb762b3fb9ac3cf8792768e8ba1f WatchSource:0}: Error finding container dffee8bdfb7f2d7e557dbe805797aadba9dadb762b3fb9ac3cf8792768e8ba1f: Status 404 returned error can't find the container with id dffee8bdfb7f2d7e557dbe805797aadba9dadb762b3fb9ac3cf8792768e8ba1f May 26 16:07:53.363399 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:07:53.363349 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" event={"ID":"0cd733a6-f301-4fc5-988e-ddc2626b4968","Type":"ContainerStarted","Data":"dffee8bdfb7f2d7e557dbe805797aadba9dadb762b3fb9ac3cf8792768e8ba1f"} May 26 16:08:00.386871 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:00.386818 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" event={"ID":"0cd733a6-f301-4fc5-988e-ddc2626b4968","Type":"ContainerStarted","Data":"f44f14a1dc0c29e2960fa4b36c1b810df30cb3200bc62eb0f6f9c318c47ca8cd"} May 26 16:08:00.387267 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:00.386997 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:08:00.406436 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:00.406391 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" podStartSLOduration=9.815349319 podStartE2EDuration="16.406378298s" podCreationTimestamp="2026-05-26 16:07:44 +0000 UTC" firstStartedPulling="2026-05-26 16:07:52.779550195 +0000 UTC m=+421.639472648" lastFinishedPulling="2026-05-26 16:07:59.370579178 +0000 UTC m=+428.230501627" observedRunningTime="2026-05-26 16:08:00.403866919 +0000 UTC m=+429.263789385" watchObservedRunningTime="2026-05-26 16:08:00.406378298 +0000 UTC m=+429.266300767" May 26 16:08:06.338493 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:06.338459 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-wtfmk" May 26 16:08:10.356105 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:10.356072 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-9cpnv" May 26 16:08:21.392722 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:21.392648 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-7z7vw" May 26 16:08:36.775395 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.775347 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh"] May 26 16:08:36.781254 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.781231 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:36.784054 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.784024 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" May 26 16:08:36.784054 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.784024 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" May 26 16:08:36.784907 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.784884 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dt9sc\"" May 26 16:08:36.787661 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.787639 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh"] May 26 16:08:36.815792 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.815767 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:36.815942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.815895 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:36.815942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.815932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7fw\" (UniqueName: \"kubernetes.io/projected/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-kube-api-access-hp7fw\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:36.917105 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.917070 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:36.917105 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.917105 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7fw\" (UniqueName: \"kubernetes.io/projected/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-kube-api-access-hp7fw\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:36.917416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.917139 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:36.917532 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.917510 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:36.917592 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.917517 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:36.925337 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:36.925317 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7fw\" (UniqueName: \"kubernetes.io/projected/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-kube-api-access-hp7fw\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:37.091285 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:37.091202 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:37.221541 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:37.221517 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh"] May 26 16:08:37.223684 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:08:37.223652 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df0d5e3_f53c_4cfa_8827_7eeb5fd06e80.slice/crio-3aaf8853158a47e9c8dfe78a2e2ccea94c67b239553ff605600a6d4a6ebb9d42 WatchSource:0}: Error finding container 3aaf8853158a47e9c8dfe78a2e2ccea94c67b239553ff605600a6d4a6ebb9d42: Status 404 returned error can't find the container with id 3aaf8853158a47e9c8dfe78a2e2ccea94c67b239553ff605600a6d4a6ebb9d42 May 26 16:08:37.512142 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:37.512053 2573 generic.go:358] "Generic (PLEG): container finished" podID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerID="02961b2de71b71ae90d0a0f5a53baa146134921e65ab12c9cbf35abea032e03a" exitCode=0 May 26 16:08:37.512142 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:37.512102 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" event={"ID":"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80","Type":"ContainerDied","Data":"02961b2de71b71ae90d0a0f5a53baa146134921e65ab12c9cbf35abea032e03a"} May 26 16:08:37.512142 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:37.512142 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" event={"ID":"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80","Type":"ContainerStarted","Data":"3aaf8853158a47e9c8dfe78a2e2ccea94c67b239553ff605600a6d4a6ebb9d42"} May 26 16:08:39.520328 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:39.520292 2573 generic.go:358] "Generic (PLEG): container finished" podID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerID="214a6a2100e9a3c95f8c5829518317e20d51af6ec1fd1e350c8120b7dbc68148" exitCode=0 May 26 16:08:39.520732 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:39.520382 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" event={"ID":"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80","Type":"ContainerDied","Data":"214a6a2100e9a3c95f8c5829518317e20d51af6ec1fd1e350c8120b7dbc68148"} May 26 16:08:40.525461 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:40.525421 2573 generic.go:358] "Generic (PLEG): container finished" podID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerID="a87c685eb2480cca012b09abf51d512be153d44d3a70ac410074a1cd9a0f93ad" exitCode=0 May 26 16:08:40.525859 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:40.525511 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" event={"ID":"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80","Type":"ContainerDied","Data":"a87c685eb2480cca012b09abf51d512be153d44d3a70ac410074a1cd9a0f93ad"} May 26 16:08:41.657456 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.657434 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:41.754314 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.754287 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp7fw\" (UniqueName: \"kubernetes.io/projected/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-kube-api-access-hp7fw\") pod \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " May 26 16:08:41.754453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.754359 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-util\") pod \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " May 26 16:08:41.754453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.754401 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-bundle\") pod \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\" (UID: \"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80\") " May 26 16:08:41.755112 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.755084 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-bundle" (OuterVolumeSpecName: "bundle") pod "7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" (UID: "7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:08:41.756320 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.756299 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-kube-api-access-hp7fw" (OuterVolumeSpecName: "kube-api-access-hp7fw") pod "7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" (UID: "7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80"). InnerVolumeSpecName "kube-api-access-hp7fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:08:41.762632 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.762590 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-util" (OuterVolumeSpecName: "util") pod "7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" (UID: "7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:08:41.855852 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.855746 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp7fw\" (UniqueName: \"kubernetes.io/projected/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-kube-api-access-hp7fw\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:08:41.855852 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.855784 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-util\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:08:41.855852 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:41.855815 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:08:42.533910 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:42.533872 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" event={"ID":"7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80","Type":"ContainerDied","Data":"3aaf8853158a47e9c8dfe78a2e2ccea94c67b239553ff605600a6d4a6ebb9d42"} May 26 16:08:42.533910 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:42.533899 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dmpdbh" May 26 16:08:42.534108 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:42.533904 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aaf8853158a47e9c8dfe78a2e2ccea94c67b239553ff605600a6d4a6ebb9d42" May 26 16:08:51.489379 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.489343 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-988cdc694-h4mq5"] May 26 16:08:51.489884 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.489866 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerName="extract" May 26 16:08:51.489951 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.489886 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerName="extract" May 26 16:08:51.489951 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.489905 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerName="pull" May 26 16:08:51.489951 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.489911 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerName="pull" May 26 16:08:51.489951 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.489921 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerName="util" May 26 16:08:51.489951 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.489927 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerName="util" May 26 16:08:51.490121 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.489983 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="7df0d5e3-f53c-4cfa-8827-7eeb5fd06e80" containerName="extract" May 26 16:08:51.492041 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.492021 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.506699 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.506677 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-988cdc694-h4mq5"] May 26 16:08:51.638588 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.638554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-trusted-ca-bundle\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.638783 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.638602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a24e1b4-cb4e-444f-838a-91327757542b-console-oauth-config\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.638783 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.638624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-oauth-serving-cert\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.638783 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.638706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-service-ca\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.638783 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.638757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-console-config\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.638988 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.638782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4gcn\" (UniqueName: \"kubernetes.io/projected/6a24e1b4-cb4e-444f-838a-91327757542b-kube-api-access-t4gcn\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.638988 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.638866 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a24e1b4-cb4e-444f-838a-91327757542b-console-serving-cert\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.740373 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.740286 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-service-ca\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.740373 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.740326 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-console-config\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.740373 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.740347 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t4gcn\" (UniqueName: \"kubernetes.io/projected/6a24e1b4-cb4e-444f-838a-91327757542b-kube-api-access-t4gcn\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.740623 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.740378 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a24e1b4-cb4e-444f-838a-91327757542b-console-serving-cert\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.740623 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.740416 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-trusted-ca-bundle\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.740623 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.740595 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a24e1b4-cb4e-444f-838a-91327757542b-console-oauth-config\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.740775 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.740649 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-oauth-serving-cert\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.740998 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.740973 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-service-ca\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.741145 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.741122 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-console-config\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.741288 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.741262 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-oauth-serving-cert\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.741540 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.741518 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a24e1b4-cb4e-444f-838a-91327757542b-trusted-ca-bundle\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.743230 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.743202 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a24e1b4-cb4e-444f-838a-91327757542b-console-serving-cert\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.743329 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.743313 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a24e1b4-cb4e-444f-838a-91327757542b-console-oauth-config\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.754311 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.754288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4gcn\" (UniqueName: \"kubernetes.io/projected/6a24e1b4-cb4e-444f-838a-91327757542b-kube-api-access-t4gcn\") pod \"console-988cdc694-h4mq5\" (UID: \"6a24e1b4-cb4e-444f-838a-91327757542b\") " pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.801860 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.801792 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-988cdc694-h4mq5" May 26 16:08:51.963870 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:51.963845 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-988cdc694-h4mq5"] May 26 16:08:51.966129 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:08:51.966100 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a24e1b4_cb4e_444f_838a_91327757542b.slice/crio-2cfee343e397594c9bcba82853a6300b38733acec7aebffbda7daed7253dccc5 WatchSource:0}: Error finding container 2cfee343e397594c9bcba82853a6300b38733acec7aebffbda7daed7253dccc5: Status 404 returned error can't find the container with id 2cfee343e397594c9bcba82853a6300b38733acec7aebffbda7daed7253dccc5 May 26 16:08:52.572230 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:52.572200 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-988cdc694-h4mq5" event={"ID":"6a24e1b4-cb4e-444f-838a-91327757542b","Type":"ContainerStarted","Data":"1609f79c758c625c4caea96d57e6f24dd67a8a9ac880be8279ebf55f06231378"} May 26 16:08:52.572230 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:52.572234 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-988cdc694-h4mq5" event={"ID":"6a24e1b4-cb4e-444f-838a-91327757542b","Type":"ContainerStarted","Data":"2cfee343e397594c9bcba82853a6300b38733acec7aebffbda7daed7253dccc5"} May 26 16:08:52.604786 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:52.604736 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-988cdc694-h4mq5" podStartSLOduration=1.604719703 podStartE2EDuration="1.604719703s" podCreationTimestamp="2026-05-26 16:08:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:08:52.603019521 +0000 UTC m=+481.462941990" watchObservedRunningTime="2026-05-26 16:08:52.604719703 +0000 UTC m=+481.464642172" May 26 16:08:59.100995 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.100962 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf"] May 26 16:08:59.103529 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.103511 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.105980 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.105958 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dt9sc\"" May 26 16:08:59.106096 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.106007 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" May 26 16:08:59.106790 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.106770 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" May 26 16:08:59.112222 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.112202 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf"] May 26 16:08:59.205303 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.205252 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.205488 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.205320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.205488 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.205434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc5dx\" (UniqueName: \"kubernetes.io/projected/6cf34b5a-666f-4110-9773-2d708dcd0627-kube-api-access-fc5dx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.306123 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.306066 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fc5dx\" (UniqueName: \"kubernetes.io/projected/6cf34b5a-666f-4110-9773-2d708dcd0627-kube-api-access-fc5dx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.306341 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.306141 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.306341 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.306166 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.306539 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.306520 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.306588 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.306547 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.314723 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.314701 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc5dx\" (UniqueName: \"kubernetes.io/projected/6cf34b5a-666f-4110-9773-2d708dcd0627-kube-api-access-fc5dx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.414543 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.414457 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:08:59.540219 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.540191 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf"] May 26 16:08:59.542980 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:08:59.542954 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf34b5a_666f_4110_9773_2d708dcd0627.slice/crio-0a6e760eabb72866aaed364bae13e351f19ced313e0127d4dab7673a72d029ed WatchSource:0}: Error finding container 0a6e760eabb72866aaed364bae13e351f19ced313e0127d4dab7673a72d029ed: Status 404 returned error can't find the container with id 0a6e760eabb72866aaed364bae13e351f19ced313e0127d4dab7673a72d029ed May 26 16:08:59.597389 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:08:59.597357 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" event={"ID":"6cf34b5a-666f-4110-9773-2d708dcd0627","Type":"ContainerStarted","Data":"0a6e760eabb72866aaed364bae13e351f19ced313e0127d4dab7673a72d029ed"} May 26 16:09:00.602387 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:00.602276 2573 generic.go:358] "Generic (PLEG): container finished" podID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerID="c77618516d4ce83aa07bc3408f6082167688b0d8b10f52d0a8a43a9f966ab502" exitCode=0 May 26 16:09:00.602387 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:00.602359 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" event={"ID":"6cf34b5a-666f-4110-9773-2d708dcd0627","Type":"ContainerDied","Data":"c77618516d4ce83aa07bc3408f6082167688b0d8b10f52d0a8a43a9f966ab502"} May 26 16:09:01.802973 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:01.802938 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-988cdc694-h4mq5" May 26 16:09:01.803450 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:01.802986 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-988cdc694-h4mq5" May 26 16:09:01.808538 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:01.808510 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-988cdc694-h4mq5" May 26 16:09:02.613692 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:02.613665 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-988cdc694-h4mq5" May 26 16:09:02.664207 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:02.664176 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c6c47dcd7-qbzfk"] May 26 16:09:03.614480 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:03.614445 2573 generic.go:358] "Generic (PLEG): container finished" podID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerID="36f6ff0cb64eb37526ad732c28cd7cd8ee62b436c4541685f98b4c12133d44ba" exitCode=0 May 26 16:09:03.614878 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:03.614530 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" event={"ID":"6cf34b5a-666f-4110-9773-2d708dcd0627","Type":"ContainerDied","Data":"36f6ff0cb64eb37526ad732c28cd7cd8ee62b436c4541685f98b4c12133d44ba"} May 26 16:09:04.620186 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:04.620147 2573 generic.go:358] "Generic (PLEG): container finished" podID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerID="1a4793b87dcbf0636e4826f48142a08c73baed59e5ada96891946ffa136bf88e" exitCode=0 May 26 16:09:04.620593 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:04.620240 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" event={"ID":"6cf34b5a-666f-4110-9773-2d708dcd0627","Type":"ContainerDied","Data":"1a4793b87dcbf0636e4826f48142a08c73baed59e5ada96891946ffa136bf88e"} May 26 16:09:05.747037 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.747012 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:09:05.859704 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.859665 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-bundle\") pod \"6cf34b5a-666f-4110-9773-2d708dcd0627\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " May 26 16:09:05.859937 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.859720 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc5dx\" (UniqueName: \"kubernetes.io/projected/6cf34b5a-666f-4110-9773-2d708dcd0627-kube-api-access-fc5dx\") pod \"6cf34b5a-666f-4110-9773-2d708dcd0627\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " May 26 16:09:05.859937 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.859798 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-util\") pod \"6cf34b5a-666f-4110-9773-2d708dcd0627\" (UID: \"6cf34b5a-666f-4110-9773-2d708dcd0627\") " May 26 16:09:05.860177 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.860148 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-bundle" (OuterVolumeSpecName: "bundle") pod "6cf34b5a-666f-4110-9773-2d708dcd0627" (UID: "6cf34b5a-666f-4110-9773-2d708dcd0627"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:09:05.861883 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.861856 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf34b5a-666f-4110-9773-2d708dcd0627-kube-api-access-fc5dx" (OuterVolumeSpecName: "kube-api-access-fc5dx") pod "6cf34b5a-666f-4110-9773-2d708dcd0627" (UID: "6cf34b5a-666f-4110-9773-2d708dcd0627"). InnerVolumeSpecName "kube-api-access-fc5dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:09:05.864141 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.864116 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-util" (OuterVolumeSpecName: "util") pod "6cf34b5a-666f-4110-9773-2d708dcd0627" (UID: "6cf34b5a-666f-4110-9773-2d708dcd0627"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:09:05.960920 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.960881 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc5dx\" (UniqueName: \"kubernetes.io/projected/6cf34b5a-666f-4110-9773-2d708dcd0627-kube-api-access-fc5dx\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:05.960920 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.960911 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-util\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:05.960920 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:05.960921 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cf34b5a-666f-4110-9773-2d708dcd0627-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:06.628085 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:06.628052 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" May 26 16:09:06.628261 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:06.628051 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwlhlf" event={"ID":"6cf34b5a-666f-4110-9773-2d708dcd0627","Type":"ContainerDied","Data":"0a6e760eabb72866aaed364bae13e351f19ced313e0127d4dab7673a72d029ed"} May 26 16:09:06.628261 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:06.628162 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a6e760eabb72866aaed364bae13e351f19ced313e0127d4dab7673a72d029ed" May 26 16:09:11.942143 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.942107 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb"] May 26 16:09:11.942602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.942467 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerName="extract" May 26 16:09:11.942602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.942477 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerName="extract" May 26 16:09:11.942602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.942505 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerName="util" May 26 16:09:11.942602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.942511 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerName="util" May 26 16:09:11.942602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.942517 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerName="pull" May 26 16:09:11.942602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.942524 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerName="pull" May 26 16:09:11.942602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.942580 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="6cf34b5a-666f-4110-9773-2d708dcd0627" containerName="extract" May 26 16:09:11.945708 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.945691 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" May 26 16:09:11.948089 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.948067 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" May 26 16:09:11.948760 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.948742 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pd9c2\"" May 26 16:09:11.948883 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.948743 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" May 26 16:09:11.955013 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:11.954994 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb"] May 26 16:09:12.119238 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:12.119198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f660637-c4c2-42fa-ada9-c8b59e42b5a1-tmp\") pod \"openshift-lws-operator-bfc7f696d-j5hlb\" (UID: \"0f660637-c4c2-42fa-ada9-c8b59e42b5a1\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" May 26 16:09:12.119422 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:12.119246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vjv\" (UniqueName: \"kubernetes.io/projected/0f660637-c4c2-42fa-ada9-c8b59e42b5a1-kube-api-access-c8vjv\") pod \"openshift-lws-operator-bfc7f696d-j5hlb\" (UID: \"0f660637-c4c2-42fa-ada9-c8b59e42b5a1\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" May 26 16:09:12.220296 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:12.220215 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f660637-c4c2-42fa-ada9-c8b59e42b5a1-tmp\") pod \"openshift-lws-operator-bfc7f696d-j5hlb\" (UID: \"0f660637-c4c2-42fa-ada9-c8b59e42b5a1\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" May 26 16:09:12.220296 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:12.220262 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vjv\" (UniqueName: \"kubernetes.io/projected/0f660637-c4c2-42fa-ada9-c8b59e42b5a1-kube-api-access-c8vjv\") pod \"openshift-lws-operator-bfc7f696d-j5hlb\" (UID: \"0f660637-c4c2-42fa-ada9-c8b59e42b5a1\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" May 26 16:09:12.220618 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:12.220594 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f660637-c4c2-42fa-ada9-c8b59e42b5a1-tmp\") pod \"openshift-lws-operator-bfc7f696d-j5hlb\" (UID: \"0f660637-c4c2-42fa-ada9-c8b59e42b5a1\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" May 26 16:09:12.228381 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:12.228352 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vjv\" (UniqueName: \"kubernetes.io/projected/0f660637-c4c2-42fa-ada9-c8b59e42b5a1-kube-api-access-c8vjv\") pod \"openshift-lws-operator-bfc7f696d-j5hlb\" (UID: \"0f660637-c4c2-42fa-ada9-c8b59e42b5a1\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" May 26 16:09:12.262222 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:12.262195 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" May 26 16:09:12.391269 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:12.391239 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb"] May 26 16:09:12.393157 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:09:12.393126 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f660637_c4c2_42fa_ada9_c8b59e42b5a1.slice/crio-bf150746180f37b67b061c4f9b5b610a73a28ffd13fa68f34f78b30d0f6608ab WatchSource:0}: Error finding container bf150746180f37b67b061c4f9b5b610a73a28ffd13fa68f34f78b30d0f6608ab: Status 404 returned error can't find the container with id bf150746180f37b67b061c4f9b5b610a73a28ffd13fa68f34f78b30d0f6608ab May 26 16:09:12.650657 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:12.650624 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" event={"ID":"0f660637-c4c2-42fa-ada9-c8b59e42b5a1","Type":"ContainerStarted","Data":"bf150746180f37b67b061c4f9b5b610a73a28ffd13fa68f34f78b30d0f6608ab"} May 26 16:09:15.663266 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:15.663224 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" event={"ID":"0f660637-c4c2-42fa-ada9-c8b59e42b5a1","Type":"ContainerStarted","Data":"68b86855142625aa509e88f8d38aebc0492e5e6e8dcefd5f00e28ddaa7efaa07"} May 26 16:09:15.682543 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:15.682494 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-j5hlb" podStartSLOduration=2.308102966 podStartE2EDuration="4.682478509s" podCreationTimestamp="2026-05-26 16:09:11 +0000 UTC" firstStartedPulling="2026-05-26 16:09:12.394562989 +0000 UTC m=+501.254485440" lastFinishedPulling="2026-05-26 16:09:14.768938521 +0000 UTC m=+503.628860983" observedRunningTime="2026-05-26 16:09:15.680311096 +0000 UTC m=+504.540233564" watchObservedRunningTime="2026-05-26 16:09:15.682478509 +0000 UTC m=+504.542400975" May 26 16:09:27.686361 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.686303 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5c6c47dcd7-qbzfk" podUID="6fb901b9-c7d8-49cf-99e5-04e1f19c8031" containerName="console" containerID="cri-o://7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9" gracePeriod=15 May 26 16:09:27.698713 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.698613 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x"] May 26 16:09:27.701607 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.701590 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.704045 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.704028 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" May 26 16:09:27.704738 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.704722 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dt9sc\"" May 26 16:09:27.704820 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.704751 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" May 26 16:09:27.709249 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.709224 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x"] May 26 16:09:27.734022 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.733998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.734152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.734107 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.734152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.734138 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxbj\" (UniqueName: \"kubernetes.io/projected/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-kube-api-access-tlxbj\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.803314 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.803287 2573 patch_prober.go:28] interesting pod/console-5c6c47dcd7-qbzfk container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.25:8443/health\": dial tcp 10.133.0.25:8443: connect: connection refused" start-of-body= May 26 16:09:27.803393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.803361 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-5c6c47dcd7-qbzfk" podUID="6fb901b9-c7d8-49cf-99e5-04e1f19c8031" containerName="console" probeResult="failure" output="Get \"https://10.133.0.25:8443/health\": dial tcp 10.133.0.25:8443: connect: connection refused" May 26 16:09:27.834495 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.834457 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.834683 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.834569 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.834683 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.834603 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxbj\" (UniqueName: \"kubernetes.io/projected/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-kube-api-access-tlxbj\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.834862 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.834824 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.834933 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.834897 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.843301 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.843276 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxbj\" (UniqueName: \"kubernetes.io/projected/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-kube-api-access-tlxbj\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:27.934083 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.934060 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c6c47dcd7-qbzfk_6fb901b9-c7d8-49cf-99e5-04e1f19c8031/console/0.log" May 26 16:09:27.934184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.934121 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:09:27.935148 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935132 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-serving-cert\") pod \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " May 26 16:09:27.935199 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935163 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-oauth-serving-cert\") pod \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " May 26 16:09:27.935239 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935205 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-trusted-ca-bundle\") pod \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " May 26 16:09:27.935239 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935235 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-service-ca\") pod \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " May 26 16:09:27.935338 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935278 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-config\") pod \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " May 26 16:09:27.935338 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935310 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99262\" (UniqueName: \"kubernetes.io/projected/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-kube-api-access-99262\") pod \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " May 26 16:09:27.935442 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935337 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-oauth-config\") pod \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\" (UID: \"6fb901b9-c7d8-49cf-99e5-04e1f19c8031\") " May 26 16:09:27.935719 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935676 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6fb901b9-c7d8-49cf-99e5-04e1f19c8031" (UID: "6fb901b9-c7d8-49cf-99e5-04e1f19c8031"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:09:27.935719 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935672 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-service-ca" (OuterVolumeSpecName: "service-ca") pod "6fb901b9-c7d8-49cf-99e5-04e1f19c8031" (UID: "6fb901b9-c7d8-49cf-99e5-04e1f19c8031"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:09:27.935917 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935720 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6fb901b9-c7d8-49cf-99e5-04e1f19c8031" (UID: "6fb901b9-c7d8-49cf-99e5-04e1f19c8031"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:09:27.935917 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.935730 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-config" (OuterVolumeSpecName: "console-config") pod "6fb901b9-c7d8-49cf-99e5-04e1f19c8031" (UID: "6fb901b9-c7d8-49cf-99e5-04e1f19c8031"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 26 16:09:27.937303 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.937245 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6fb901b9-c7d8-49cf-99e5-04e1f19c8031" (UID: "6fb901b9-c7d8-49cf-99e5-04e1f19c8031"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:09:27.937480 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.937453 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6fb901b9-c7d8-49cf-99e5-04e1f19c8031" (UID: "6fb901b9-c7d8-49cf-99e5-04e1f19c8031"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:09:27.937535 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:27.937517 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-kube-api-access-99262" (OuterVolumeSpecName: "kube-api-access-99262") pod "6fb901b9-c7d8-49cf-99e5-04e1f19c8031" (UID: "6fb901b9-c7d8-49cf-99e5-04e1f19c8031"). InnerVolumeSpecName "kube-api-access-99262". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:09:28.037144 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.037107 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-config\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:28.037144 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.037143 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99262\" (UniqueName: \"kubernetes.io/projected/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-kube-api-access-99262\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:28.037348 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.037154 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-oauth-config\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:28.037348 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.037165 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-console-serving-cert\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:28.037348 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.037174 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-oauth-serving-cert\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:28.037348 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.037183 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-trusted-ca-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:28.037348 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.037191 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb901b9-c7d8-49cf-99e5-04e1f19c8031-service-ca\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:28.055005 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.054983 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:28.188433 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.188411 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x"] May 26 16:09:28.190350 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:09:28.190319 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25c7539_1474_4ecf_b978_b0a5f0ed9d83.slice/crio-5d66455ab402957820868ba98ee3928cea40bea28a96ca39f23f68174b6a86f7 WatchSource:0}: Error finding container 5d66455ab402957820868ba98ee3928cea40bea28a96ca39f23f68174b6a86f7: Status 404 returned error can't find the container with id 5d66455ab402957820868ba98ee3928cea40bea28a96ca39f23f68174b6a86f7 May 26 16:09:28.713877 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.713845 2573 generic.go:358] "Generic (PLEG): container finished" podID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerID="d3a041144e012b732f9cd2ce853cccf16874758f5c7209be13916742324d33cd" exitCode=0 May 26 16:09:28.714274 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.713898 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" event={"ID":"e25c7539-1474-4ecf-b978-b0a5f0ed9d83","Type":"ContainerDied","Data":"d3a041144e012b732f9cd2ce853cccf16874758f5c7209be13916742324d33cd"} May 26 16:09:28.714274 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.713939 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" event={"ID":"e25c7539-1474-4ecf-b978-b0a5f0ed9d83","Type":"ContainerStarted","Data":"5d66455ab402957820868ba98ee3928cea40bea28a96ca39f23f68174b6a86f7"} May 26 16:09:28.715231 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.715211 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c6c47dcd7-qbzfk_6fb901b9-c7d8-49cf-99e5-04e1f19c8031/console/0.log" May 26 16:09:28.715328 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.715246 2573 generic.go:358] "Generic (PLEG): container finished" podID="6fb901b9-c7d8-49cf-99e5-04e1f19c8031" containerID="7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9" exitCode=2 May 26 16:09:28.715328 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.715310 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6c47dcd7-qbzfk" May 26 16:09:28.715451 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.715328 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6c47dcd7-qbzfk" event={"ID":"6fb901b9-c7d8-49cf-99e5-04e1f19c8031","Type":"ContainerDied","Data":"7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9"} May 26 16:09:28.715451 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.715359 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6c47dcd7-qbzfk" event={"ID":"6fb901b9-c7d8-49cf-99e5-04e1f19c8031","Type":"ContainerDied","Data":"2d8f2bc4212745e6d6187cf65a4e5fb34f82fd3e8a040198e04d05b0f38d1aca"} May 26 16:09:28.715451 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.715379 2573 scope.go:117] "RemoveContainer" containerID="7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9" May 26 16:09:28.723884 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.723867 2573 scope.go:117] "RemoveContainer" containerID="7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9" May 26 16:09:28.724319 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:09:28.724297 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9\": container with ID starting with 7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9 not found: ID does not exist" containerID="7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9" May 26 16:09:28.724393 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.724330 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9"} err="failed to get container status \"7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9\": rpc error: code = NotFound desc = could not find container \"7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9\": container with ID starting with 7d220ccc12d1c09050ceccea0ae98045e2400e4aa9525de21fb0c9e3148412d9 not found: ID does not exist" May 26 16:09:28.749222 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.749190 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c6c47dcd7-qbzfk"] May 26 16:09:28.755655 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:28.755632 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c6c47dcd7-qbzfk"] May 26 16:09:29.874546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:29.874513 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fb901b9-c7d8-49cf-99e5-04e1f19c8031" path="/var/lib/kubelet/pods/6fb901b9-c7d8-49cf-99e5-04e1f19c8031/volumes" May 26 16:09:30.730882 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:30.730824 2573 generic.go:358] "Generic (PLEG): container finished" podID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerID="9ff486089ef5fd6b42a12773316c93d5038f190cf506b727a2332afda28dcb66" exitCode=0 May 26 16:09:30.731077 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:30.730857 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" event={"ID":"e25c7539-1474-4ecf-b978-b0a5f0ed9d83","Type":"ContainerDied","Data":"9ff486089ef5fd6b42a12773316c93d5038f190cf506b727a2332afda28dcb66"} May 26 16:09:31.736194 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:31.736160 2573 generic.go:358] "Generic (PLEG): container finished" podID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerID="193d6337112cfb4d1ece2566f337d16b4cd1ba29c4c505013656d7b37840332f" exitCode=0 May 26 16:09:31.736568 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:31.736253 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" event={"ID":"e25c7539-1474-4ecf-b978-b0a5f0ed9d83","Type":"ContainerDied","Data":"193d6337112cfb4d1ece2566f337d16b4cd1ba29c4c505013656d7b37840332f"} May 26 16:09:32.871344 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:32.871320 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:32.977470 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:32.977432 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-util\") pod \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " May 26 16:09:32.977616 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:32.977528 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlxbj\" (UniqueName: \"kubernetes.io/projected/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-kube-api-access-tlxbj\") pod \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " May 26 16:09:32.977616 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:32.977578 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-bundle\") pod \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\" (UID: \"e25c7539-1474-4ecf-b978-b0a5f0ed9d83\") " May 26 16:09:32.978583 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:32.978561 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-bundle" (OuterVolumeSpecName: "bundle") pod "e25c7539-1474-4ecf-b978-b0a5f0ed9d83" (UID: "e25c7539-1474-4ecf-b978-b0a5f0ed9d83"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:09:32.979760 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:32.979739 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-kube-api-access-tlxbj" (OuterVolumeSpecName: "kube-api-access-tlxbj") pod "e25c7539-1474-4ecf-b978-b0a5f0ed9d83" (UID: "e25c7539-1474-4ecf-b978-b0a5f0ed9d83"). InnerVolumeSpecName "kube-api-access-tlxbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:09:32.983128 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:32.983108 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-util" (OuterVolumeSpecName: "util") pod "e25c7539-1474-4ecf-b978-b0a5f0ed9d83" (UID: "e25c7539-1474-4ecf-b978-b0a5f0ed9d83"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:09:33.078172 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:33.078139 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxbj\" (UniqueName: \"kubernetes.io/projected/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-kube-api-access-tlxbj\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:33.078172 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:33.078168 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:33.078172 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:33.078181 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25c7539-1474-4ecf-b978-b0a5f0ed9d83-util\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:33.744096 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:33.744060 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" event={"ID":"e25c7539-1474-4ecf-b978-b0a5f0ed9d83","Type":"ContainerDied","Data":"5d66455ab402957820868ba98ee3928cea40bea28a96ca39f23f68174b6a86f7"} May 26 16:09:33.744096 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:33.744096 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c48354s78x" May 26 16:09:33.744314 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:33.744099 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d66455ab402957820868ba98ee3928cea40bea28a96ca39f23f68174b6a86f7" May 26 16:09:42.909534 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.909502 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g"] May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.909872 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerName="pull" May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.909883 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerName="pull" May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.909894 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerName="util" May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.909900 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerName="util" May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.909930 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerName="extract" May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.909935 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerName="extract" May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.909942 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fb901b9-c7d8-49cf-99e5-04e1f19c8031" containerName="console" May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.909947 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fb901b9-c7d8-49cf-99e5-04e1f19c8031" containerName="console" May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.910009 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="e25c7539-1474-4ecf-b978-b0a5f0ed9d83" containerName="extract" May 26 16:09:42.910027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.910017 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="6fb901b9-c7d8-49cf-99e5-04e1f19c8031" containerName="console" May 26 16:09:42.912062 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.912046 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:42.915269 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.915252 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" May 26 16:09:42.915634 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.915618 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" May 26 16:09:42.916207 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.916192 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dt9sc\"" May 26 16:09:42.933019 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.932998 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g"] May 26 16:09:42.962080 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.962050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:42.962235 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.962100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz6wl\" (UniqueName: \"kubernetes.io/projected/0480022d-7882-4254-8cac-0ff7bcce64ed-kube-api-access-zz6wl\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:42.962277 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:42.962219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:43.062793 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.062755 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:43.062993 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.062864 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:43.062993 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.062933 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zz6wl\" (UniqueName: \"kubernetes.io/projected/0480022d-7882-4254-8cac-0ff7bcce64ed-kube-api-access-zz6wl\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:43.063168 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.063147 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:43.063229 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.063208 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:43.073293 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.073257 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz6wl\" (UniqueName: \"kubernetes.io/projected/0480022d-7882-4254-8cac-0ff7bcce64ed-kube-api-access-zz6wl\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:43.221424 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.221329 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:43.362344 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.359562 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g"] May 26 16:09:43.779471 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.779435 2573 generic.go:358] "Generic (PLEG): container finished" podID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerID="04e717e35cd47416d3bd7560ebc4302ea9e6e70c3b9109ebf299c95ff6480544" exitCode=0 May 26 16:09:43.779677 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.779521 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" event={"ID":"0480022d-7882-4254-8cac-0ff7bcce64ed","Type":"ContainerDied","Data":"04e717e35cd47416d3bd7560ebc4302ea9e6e70c3b9109ebf299c95ff6480544"} May 26 16:09:43.779677 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:43.779556 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" event={"ID":"0480022d-7882-4254-8cac-0ff7bcce64ed","Type":"ContainerStarted","Data":"899af9197937c8f85d5765af50659cc406b2c141ecf48a885f9c5c43b6233414"} May 26 16:09:44.137040 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.137007 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8"] May 26 16:09:44.139314 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.139297 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:09:44.142053 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.142020 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"openshift-service-ca.crt\"" May 26 16:09:44.142053 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.142038 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"kube-root-ca.crt\"" May 26 16:09:44.142222 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.142102 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators\"/\"servicemesh-operator3-dockercfg-pmnhw\"" May 26 16:09:44.152824 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.152800 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8"] May 26 16:09:44.174773 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.174746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/9c3dfb08-92be-4d06-861f-4ede39ba6b44-operator-config\") pod \"servicemesh-operator3-55f49c5f94-xmjf8\" (UID: \"9c3dfb08-92be-4d06-861f-4ede39ba6b44\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:09:44.174945 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.174804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwpc\" (UniqueName: \"kubernetes.io/projected/9c3dfb08-92be-4d06-861f-4ede39ba6b44-kube-api-access-9nwpc\") pod \"servicemesh-operator3-55f49c5f94-xmjf8\" (UID: \"9c3dfb08-92be-4d06-861f-4ede39ba6b44\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:09:44.276069 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.276029 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/9c3dfb08-92be-4d06-861f-4ede39ba6b44-operator-config\") pod \"servicemesh-operator3-55f49c5f94-xmjf8\" (UID: \"9c3dfb08-92be-4d06-861f-4ede39ba6b44\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:09:44.276254 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.276101 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwpc\" (UniqueName: \"kubernetes.io/projected/9c3dfb08-92be-4d06-861f-4ede39ba6b44-kube-api-access-9nwpc\") pod \"servicemesh-operator3-55f49c5f94-xmjf8\" (UID: \"9c3dfb08-92be-4d06-861f-4ede39ba6b44\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:09:44.278551 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.278532 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/9c3dfb08-92be-4d06-861f-4ede39ba6b44-operator-config\") pod \"servicemesh-operator3-55f49c5f94-xmjf8\" (UID: \"9c3dfb08-92be-4d06-861f-4ede39ba6b44\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:09:44.293069 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.293046 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwpc\" (UniqueName: \"kubernetes.io/projected/9c3dfb08-92be-4d06-861f-4ede39ba6b44-kube-api-access-9nwpc\") pod \"servicemesh-operator3-55f49c5f94-xmjf8\" (UID: \"9c3dfb08-92be-4d06-861f-4ede39ba6b44\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:09:44.449123 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.449028 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:09:44.587955 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.587925 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8"] May 26 16:09:44.600189 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:09:44.600159 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3dfb08_92be_4d06_861f_4ede39ba6b44.slice/crio-83bb54c805dfac0083e0eacdc9188f386d80d2ea39763186bcd73d26c4f25a23 WatchSource:0}: Error finding container 83bb54c805dfac0083e0eacdc9188f386d80d2ea39763186bcd73d26c4f25a23: Status 404 returned error can't find the container with id 83bb54c805dfac0083e0eacdc9188f386d80d2ea39763186bcd73d26c4f25a23 May 26 16:09:44.784807 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.784772 2573 generic.go:358] "Generic (PLEG): container finished" podID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerID="c32514daac5e68e571a219175a33be79887d70c7d5cb0d24850fa1323d0df86d" exitCode=0 May 26 16:09:44.785009 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.784805 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" event={"ID":"0480022d-7882-4254-8cac-0ff7bcce64ed","Type":"ContainerDied","Data":"c32514daac5e68e571a219175a33be79887d70c7d5cb0d24850fa1323d0df86d"} May 26 16:09:44.786200 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:44.786164 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" event={"ID":"9c3dfb08-92be-4d06-861f-4ede39ba6b44","Type":"ContainerStarted","Data":"83bb54c805dfac0083e0eacdc9188f386d80d2ea39763186bcd73d26c4f25a23"} May 26 16:09:45.791486 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:45.791449 2573 generic.go:358] "Generic (PLEG): container finished" podID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerID="0a0b78c109c976a70256a68299480a37c20d55317ec07bf00a02a75d654c4578" exitCode=0 May 26 16:09:45.791875 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:45.791533 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" event={"ID":"0480022d-7882-4254-8cac-0ff7bcce64ed","Type":"ContainerDied","Data":"0a0b78c109c976a70256a68299480a37c20d55317ec07bf00a02a75d654c4578"} May 26 16:09:46.954765 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:46.954738 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:46.998084 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:46.998044 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-util\") pod \"0480022d-7882-4254-8cac-0ff7bcce64ed\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " May 26 16:09:46.998263 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:46.998112 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-bundle\") pod \"0480022d-7882-4254-8cac-0ff7bcce64ed\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " May 26 16:09:46.998263 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:46.998171 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz6wl\" (UniqueName: \"kubernetes.io/projected/0480022d-7882-4254-8cac-0ff7bcce64ed-kube-api-access-zz6wl\") pod \"0480022d-7882-4254-8cac-0ff7bcce64ed\" (UID: \"0480022d-7882-4254-8cac-0ff7bcce64ed\") " May 26 16:09:46.999232 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:46.999206 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-bundle" (OuterVolumeSpecName: "bundle") pod "0480022d-7882-4254-8cac-0ff7bcce64ed" (UID: "0480022d-7882-4254-8cac-0ff7bcce64ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:09:47.001072 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:47.001039 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0480022d-7882-4254-8cac-0ff7bcce64ed-kube-api-access-zz6wl" (OuterVolumeSpecName: "kube-api-access-zz6wl") pod "0480022d-7882-4254-8cac-0ff7bcce64ed" (UID: "0480022d-7882-4254-8cac-0ff7bcce64ed"). InnerVolumeSpecName "kube-api-access-zz6wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:09:47.007221 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:47.007172 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-util" (OuterVolumeSpecName: "util") pod "0480022d-7882-4254-8cac-0ff7bcce64ed" (UID: "0480022d-7882-4254-8cac-0ff7bcce64ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:09:47.099491 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:47.099406 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz6wl\" (UniqueName: \"kubernetes.io/projected/0480022d-7882-4254-8cac-0ff7bcce64ed-kube-api-access-zz6wl\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:47.099491 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:47.099441 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-util\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:47.099491 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:47.099456 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0480022d-7882-4254-8cac-0ff7bcce64ed-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:09:47.803187 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:47.803148 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" event={"ID":"0480022d-7882-4254-8cac-0ff7bcce64ed","Type":"ContainerDied","Data":"899af9197937c8f85d5765af50659cc406b2c141ecf48a885f9c5c43b6233414"} May 26 16:09:47.803187 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:47.803189 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="899af9197937c8f85d5765af50659cc406b2c141ecf48a885f9c5c43b6233414" May 26 16:09:47.803457 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:47.803229 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a6bk5g" May 26 16:09:48.814455 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:48.814416 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" event={"ID":"9c3dfb08-92be-4d06-861f-4ede39ba6b44","Type":"ContainerStarted","Data":"50af8cb7916dc6dbad44e543e568067477f5180e261db130846aa7a65d16e3e7"} May 26 16:09:48.814857 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:48.814534 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:09:48.835304 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:48.835226 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" podStartSLOduration=1.451300683 podStartE2EDuration="4.835211678s" podCreationTimestamp="2026-05-26 16:09:44 +0000 UTC" firstStartedPulling="2026-05-26 16:09:44.602820321 +0000 UTC m=+533.462742767" lastFinishedPulling="2026-05-26 16:09:47.986731302 +0000 UTC m=+536.846653762" observedRunningTime="2026-05-26 16:09:48.834154636 +0000 UTC m=+537.694077106" watchObservedRunningTime="2026-05-26 16:09:48.835211678 +0000 UTC m=+537.695134147" May 26 16:09:49.876536 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.876499 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-669cc6566-wwl29"] May 26 16:09:49.877053 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.877029 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerName="extract" May 26 16:09:49.877053 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.877052 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerName="extract" May 26 16:09:49.877171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.877071 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerName="util" May 26 16:09:49.877171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.877079 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerName="util" May 26 16:09:49.877171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.877106 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerName="pull" May 26 16:09:49.877171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.877114 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerName="pull" May 26 16:09:49.877346 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.877188 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="0480022d-7882-4254-8cac-0ff7bcce64ed" containerName="extract" May 26 16:09:49.879603 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.879578 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:49.885462 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.885433 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dvkwk\"" May 26 16:09:49.885943 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.885925 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" May 26 16:09:49.886381 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.886358 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" May 26 16:09:49.889400 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.889380 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" May 26 16:09:49.896745 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.896250 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-669cc6566-wwl29"] May 26 16:09:49.923504 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.923482 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/af570fde-da98-443c-9544-668d15604078-manager-config\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:49.923619 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.923521 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m4xw\" (UniqueName: \"kubernetes.io/projected/af570fde-da98-443c-9544-668d15604078-kube-api-access-8m4xw\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:49.923680 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.923624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/af570fde-da98-443c-9544-668d15604078-metrics-cert\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:49.923741 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:49.923698 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af570fde-da98-443c-9544-668d15604078-cert\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.024557 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.024523 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/af570fde-da98-443c-9544-668d15604078-manager-config\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.024704 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.024570 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8m4xw\" (UniqueName: \"kubernetes.io/projected/af570fde-da98-443c-9544-668d15604078-kube-api-access-8m4xw\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.024704 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.024614 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/af570fde-da98-443c-9544-668d15604078-metrics-cert\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.024704 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.024642 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af570fde-da98-443c-9544-668d15604078-cert\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.025228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.025200 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/af570fde-da98-443c-9544-668d15604078-manager-config\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.027258 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.027230 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af570fde-da98-443c-9544-668d15604078-cert\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.027258 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.027246 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/af570fde-da98-443c-9544-668d15604078-metrics-cert\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.036127 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.036099 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m4xw\" (UniqueName: \"kubernetes.io/projected/af570fde-da98-443c-9544-668d15604078-kube-api-access-8m4xw\") pod \"lws-controller-manager-669cc6566-wwl29\" (UID: \"af570fde-da98-443c-9544-668d15604078\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.200706 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.200622 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:50.349468 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.349442 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-669cc6566-wwl29"] May 26 16:09:50.823476 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:50.823435 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" event={"ID":"af570fde-da98-443c-9544-668d15604078","Type":"ContainerStarted","Data":"97f9df307a5612b78ad4f2f28503e2d1335f57f6d15e505afb97acd8f21d0f38"} May 26 16:09:52.832814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:52.832773 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" event={"ID":"af570fde-da98-443c-9544-668d15604078","Type":"ContainerStarted","Data":"a9e6ced3715c5c55cb8f715d1b7bb165c719570ddba9da5072afa0f49abbe9f0"} May 26 16:09:52.833353 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:52.832946 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:09:52.867672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:52.867620 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" podStartSLOduration=2.1072443339999998 podStartE2EDuration="3.867605311s" podCreationTimestamp="2026-05-26 16:09:49 +0000 UTC" firstStartedPulling="2026-05-26 16:09:50.355624735 +0000 UTC m=+539.215547196" lastFinishedPulling="2026-05-26 16:09:52.115985713 +0000 UTC m=+540.975908173" observedRunningTime="2026-05-26 16:09:52.86748784 +0000 UTC m=+541.727410305" watchObservedRunningTime="2026-05-26 16:09:52.867605311 +0000 UTC m=+541.727527779" May 26 16:09:59.820225 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:09:59.820195 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-xmjf8" May 26 16:10:03.838343 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:03.838308 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-669cc6566-wwl29" May 26 16:10:04.068892 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.068849 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz"] May 26 16:10:04.078517 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.078462 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.081415 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.081389 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz"] May 26 16:10:04.082063 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.082032 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" May 26 16:10:04.082180 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.082118 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" May 26 16:10:04.082180 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.082171 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dt9sc\"" May 26 16:10:04.155528 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.155438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bcwn\" (UniqueName: \"kubernetes.io/projected/7743ce34-6323-4292-8c22-ea9995f53f40-kube-api-access-8bcwn\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.155528 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.155501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.155719 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.155549 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.169745 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.169708 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d"] May 26 16:10:04.172331 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.172313 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.182196 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.182172 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d"] May 26 16:10:04.256198 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.256165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.256391 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.256240 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc"] May 26 16:10:04.256391 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.256257 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8bcwn\" (UniqueName: \"kubernetes.io/projected/7743ce34-6323-4292-8c22-ea9995f53f40-kube-api-access-8bcwn\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.256391 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.256291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd9k8\" (UniqueName: \"kubernetes.io/projected/ebd075ef-d961-41b1-8253-85df14a4701b-kube-api-access-nd9k8\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.256391 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.256338 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.256391 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.256369 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.256638 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.256442 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.256770 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.256750 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.256807 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.256755 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.259107 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.259092 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.264322 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.264295 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bcwn\" (UniqueName: \"kubernetes.io/projected/7743ce34-6323-4292-8c22-ea9995f53f40-kube-api-access-8bcwn\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.268286 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.268261 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc"] May 26 16:10:04.357587 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.357552 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.357780 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.357599 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.357780 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.357659 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.357780 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.357687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsm9h\" (UniqueName: \"kubernetes.io/projected/3eb39284-661f-4897-99b5-765bb3aeff16-kube-api-access-xsm9h\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.357780 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.357755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.358030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.357804 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9k8\" (UniqueName: \"kubernetes.io/projected/ebd075ef-d961-41b1-8253-85df14a4701b-kube-api-access-nd9k8\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.358088 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.358051 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.358137 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.358083 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.362712 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.362688 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7"] May 26 16:10:04.365270 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.365254 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.367121 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.367097 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9k8\" (UniqueName: \"kubernetes.io/projected/ebd075ef-d961-41b1-8253-85df14a4701b-kube-api-access-nd9k8\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.374846 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.374812 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7"] May 26 16:10:04.388937 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.388917 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:04.458995 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.458899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.458995 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.458939 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.458995 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.458965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.459227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.459010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbdx7\" (UniqueName: \"kubernetes.io/projected/2a90c77c-62a9-467b-924a-ade0956bec1f-kube-api-access-pbdx7\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.459227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.459055 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xsm9h\" (UniqueName: \"kubernetes.io/projected/3eb39284-661f-4897-99b5-765bb3aeff16-kube-api-access-xsm9h\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.459227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.459090 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.459369 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.459300 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.459429 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.459412 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.468208 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.468179 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsm9h\" (UniqueName: \"kubernetes.io/projected/3eb39284-661f-4897-99b5-765bb3aeff16-kube-api-access-xsm9h\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.482125 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.482100 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:04.528115 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.528090 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz"] May 26 16:10:04.530091 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:10:04.530054 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7743ce34_6323_4292_8c22_ea9995f53f40.slice/crio-62b265380510bd6d8252e17af833b294a89654749afbfc3ecb13222fa5769c34 WatchSource:0}: Error finding container 62b265380510bd6d8252e17af833b294a89654749afbfc3ecb13222fa5769c34: Status 404 returned error can't find the container with id 62b265380510bd6d8252e17af833b294a89654749afbfc3ecb13222fa5769c34 May 26 16:10:04.559738 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.559713 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.559940 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.559762 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.559940 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.559793 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdx7\" (UniqueName: \"kubernetes.io/projected/2a90c77c-62a9-467b-924a-ade0956bec1f-kube-api-access-pbdx7\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.560425 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.560379 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.560425 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.560409 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.571416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.571375 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdx7\" (UniqueName: \"kubernetes.io/projected/2a90c77c-62a9-467b-924a-ade0956bec1f-kube-api-access-pbdx7\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.576605 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.576584 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:04.646441 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.645922 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d"] May 26 16:10:04.667680 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:10:04.667649 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd075ef_d961_41b1_8253_85df14a4701b.slice/crio-250bfe5a9df7621e871528762d65960ae25b81fc665687daefc0cb47d8b60d58 WatchSource:0}: Error finding container 250bfe5a9df7621e871528762d65960ae25b81fc665687daefc0cb47d8b60d58: Status 404 returned error can't find the container with id 250bfe5a9df7621e871528762d65960ae25b81fc665687daefc0cb47d8b60d58 May 26 16:10:04.683954 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.683930 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:04.732781 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.732754 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc"] May 26 16:10:04.735207 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:10:04.734970 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb39284_661f_4897_99b5_765bb3aeff16.slice/crio-03ab635b88753af11b863f52a994b59a614875f1ab689bd99015159985b8e886 WatchSource:0}: Error finding container 03ab635b88753af11b863f52a994b59a614875f1ab689bd99015159985b8e886: Status 404 returned error can't find the container with id 03ab635b88753af11b863f52a994b59a614875f1ab689bd99015159985b8e886 May 26 16:10:04.832361 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.832334 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7"] May 26 16:10:04.833915 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:10:04.833888 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a90c77c_62a9_467b_924a_ade0956bec1f.slice/crio-569c3833ee3fbf02e5d55b8d90cf96fe6ea1b5b5b6d8247c0d19b7ce52462c14 WatchSource:0}: Error finding container 569c3833ee3fbf02e5d55b8d90cf96fe6ea1b5b5b6d8247c0d19b7ce52462c14: Status 404 returned error can't find the container with id 569c3833ee3fbf02e5d55b8d90cf96fe6ea1b5b5b6d8247c0d19b7ce52462c14 May 26 16:10:04.880999 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.880954 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" event={"ID":"2a90c77c-62a9-467b-924a-ade0956bec1f","Type":"ContainerStarted","Data":"569c3833ee3fbf02e5d55b8d90cf96fe6ea1b5b5b6d8247c0d19b7ce52462c14"} May 26 16:10:04.882412 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.882385 2573 generic.go:358] "Generic (PLEG): container finished" podID="ebd075ef-d961-41b1-8253-85df14a4701b" containerID="0d073b15089ba1b45f67e73ac060f49916541867c08e8ad223c4d637e29962d8" exitCode=0 May 26 16:10:04.882522 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.882475 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" event={"ID":"ebd075ef-d961-41b1-8253-85df14a4701b","Type":"ContainerDied","Data":"0d073b15089ba1b45f67e73ac060f49916541867c08e8ad223c4d637e29962d8"} May 26 16:10:04.882522 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.882511 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" event={"ID":"ebd075ef-d961-41b1-8253-85df14a4701b","Type":"ContainerStarted","Data":"250bfe5a9df7621e871528762d65960ae25b81fc665687daefc0cb47d8b60d58"} May 26 16:10:04.883980 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.883961 2573 generic.go:358] "Generic (PLEG): container finished" podID="7743ce34-6323-4292-8c22-ea9995f53f40" containerID="2abd77e38a4b79ca81c5ce954c5179dc43f3f5325568cac1a7448284938b9191" exitCode=0 May 26 16:10:04.884056 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.884040 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" event={"ID":"7743ce34-6323-4292-8c22-ea9995f53f40","Type":"ContainerDied","Data":"2abd77e38a4b79ca81c5ce954c5179dc43f3f5325568cac1a7448284938b9191"} May 26 16:10:04.884149 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.884067 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" event={"ID":"7743ce34-6323-4292-8c22-ea9995f53f40","Type":"ContainerStarted","Data":"62b265380510bd6d8252e17af833b294a89654749afbfc3ecb13222fa5769c34"} May 26 16:10:04.885566 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.885547 2573 generic.go:358] "Generic (PLEG): container finished" podID="3eb39284-661f-4897-99b5-765bb3aeff16" containerID="97dff5190095b588fd984f1a58eb3164e26f51c766239048abe89de6d670f857" exitCode=0 May 26 16:10:04.885625 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.885604 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" event={"ID":"3eb39284-661f-4897-99b5-765bb3aeff16","Type":"ContainerDied","Data":"97dff5190095b588fd984f1a58eb3164e26f51c766239048abe89de6d670f857"} May 26 16:10:04.885625 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:04.885621 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" event={"ID":"3eb39284-661f-4897-99b5-765bb3aeff16","Type":"ContainerStarted","Data":"03ab635b88753af11b863f52a994b59a614875f1ab689bd99015159985b8e886"} May 26 16:10:05.890603 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:05.890512 2573 generic.go:358] "Generic (PLEG): container finished" podID="7743ce34-6323-4292-8c22-ea9995f53f40" containerID="b7883913f6517a8c1d5ad2889bc7fc7b39ec9fb5385f34ace29adecf5daadb1e" exitCode=0 May 26 16:10:05.890603 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:05.890590 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" event={"ID":"7743ce34-6323-4292-8c22-ea9995f53f40","Type":"ContainerDied","Data":"b7883913f6517a8c1d5ad2889bc7fc7b39ec9fb5385f34ace29adecf5daadb1e"} May 26 16:10:05.892436 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:05.892411 2573 generic.go:358] "Generic (PLEG): container finished" podID="3eb39284-661f-4897-99b5-765bb3aeff16" containerID="d286ddaabaf23484a1549275f53a2c68c8b38f0bcd55b9b41d0d9de78b1cd5b4" exitCode=0 May 26 16:10:05.892494 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:05.892468 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" event={"ID":"3eb39284-661f-4897-99b5-765bb3aeff16","Type":"ContainerDied","Data":"d286ddaabaf23484a1549275f53a2c68c8b38f0bcd55b9b41d0d9de78b1cd5b4"} May 26 16:10:05.893983 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:05.893853 2573 generic.go:358] "Generic (PLEG): container finished" podID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerID="44ae6f4a0f0f125df9c0a8cd429ad580207860a575b25f6f7100b0953c008ea4" exitCode=0 May 26 16:10:05.893983 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:05.893923 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" event={"ID":"2a90c77c-62a9-467b-924a-ade0956bec1f","Type":"ContainerDied","Data":"44ae6f4a0f0f125df9c0a8cd429ad580207860a575b25f6f7100b0953c008ea4"} May 26 16:10:05.895645 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:05.895615 2573 generic.go:358] "Generic (PLEG): container finished" podID="ebd075ef-d961-41b1-8253-85df14a4701b" containerID="348c279324046edb8d25dfd7f1d0c5b46ceb4c47e6f0932861ed3833527eaae2" exitCode=0 May 26 16:10:05.895729 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:05.895706 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" event={"ID":"ebd075ef-d961-41b1-8253-85df14a4701b","Type":"ContainerDied","Data":"348c279324046edb8d25dfd7f1d0c5b46ceb4c47e6f0932861ed3833527eaae2"} May 26 16:10:06.901734 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:06.901696 2573 generic.go:358] "Generic (PLEG): container finished" podID="7743ce34-6323-4292-8c22-ea9995f53f40" containerID="551e2f4316be9e34713f4886dfc8b606a8569783999f9c6b148a4c96f3753e71" exitCode=0 May 26 16:10:06.902186 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:06.901797 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" event={"ID":"7743ce34-6323-4292-8c22-ea9995f53f40","Type":"ContainerDied","Data":"551e2f4316be9e34713f4886dfc8b606a8569783999f9c6b148a4c96f3753e71"} May 26 16:10:06.903668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:06.903642 2573 generic.go:358] "Generic (PLEG): container finished" podID="3eb39284-661f-4897-99b5-765bb3aeff16" containerID="3d58af626d6fe474c7a7aa497787e1496b2e68a734d6508b209055cf83bd06ad" exitCode=0 May 26 16:10:06.903795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:06.903689 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" event={"ID":"3eb39284-661f-4897-99b5-765bb3aeff16","Type":"ContainerDied","Data":"3d58af626d6fe474c7a7aa497787e1496b2e68a734d6508b209055cf83bd06ad"} May 26 16:10:06.905185 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:06.905164 2573 generic.go:358] "Generic (PLEG): container finished" podID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerID="979654d22f891e6e3c662e7a09ae1aaa288ea995f6918a9fffb95105b3c89b0d" exitCode=0 May 26 16:10:06.905269 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:06.905233 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" event={"ID":"2a90c77c-62a9-467b-924a-ade0956bec1f","Type":"ContainerDied","Data":"979654d22f891e6e3c662e7a09ae1aaa288ea995f6918a9fffb95105b3c89b0d"} May 26 16:10:06.907098 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:06.907063 2573 generic.go:358] "Generic (PLEG): container finished" podID="ebd075ef-d961-41b1-8253-85df14a4701b" containerID="412feca5f4dd50f7cae6f283545cbb47cfdebdd187a99584e34dcce252bea748" exitCode=0 May 26 16:10:06.907179 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:06.907152 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" event={"ID":"ebd075ef-d961-41b1-8253-85df14a4701b","Type":"ContainerDied","Data":"412feca5f4dd50f7cae6f283545cbb47cfdebdd187a99584e34dcce252bea748"} May 26 16:10:07.913672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:07.913634 2573 generic.go:358] "Generic (PLEG): container finished" podID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerID="a12bfce81d92d99ca640d812feaa9aa39157a15c8ec206a0b3ffd431956eaeb6" exitCode=0 May 26 16:10:07.914091 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:07.913675 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" event={"ID":"2a90c77c-62a9-467b-924a-ade0956bec1f","Type":"ContainerDied","Data":"a12bfce81d92d99ca640d812feaa9aa39157a15c8ec206a0b3ffd431956eaeb6"} May 26 16:10:08.060942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.060920 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:08.103301 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.103275 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:08.107085 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.107065 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:08.197519 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.197424 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bcwn\" (UniqueName: \"kubernetes.io/projected/7743ce34-6323-4292-8c22-ea9995f53f40-kube-api-access-8bcwn\") pod \"7743ce34-6323-4292-8c22-ea9995f53f40\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " May 26 16:10:08.197519 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.197478 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-util\") pod \"7743ce34-6323-4292-8c22-ea9995f53f40\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " May 26 16:10:08.197519 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.197516 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-bundle\") pod \"7743ce34-6323-4292-8c22-ea9995f53f40\" (UID: \"7743ce34-6323-4292-8c22-ea9995f53f40\") " May 26 16:10:08.197772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.197620 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-util\") pod \"ebd075ef-d961-41b1-8253-85df14a4701b\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " May 26 16:10:08.197772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.197679 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd9k8\" (UniqueName: \"kubernetes.io/projected/ebd075ef-d961-41b1-8253-85df14a4701b-kube-api-access-nd9k8\") pod \"ebd075ef-d961-41b1-8253-85df14a4701b\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " May 26 16:10:08.197772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.197725 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-bundle\") pod \"ebd075ef-d961-41b1-8253-85df14a4701b\" (UID: \"ebd075ef-d961-41b1-8253-85df14a4701b\") " May 26 16:10:08.197772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.197761 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-bundle\") pod \"3eb39284-661f-4897-99b5-765bb3aeff16\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " May 26 16:10:08.198018 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.197790 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-util\") pod \"3eb39284-661f-4897-99b5-765bb3aeff16\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " May 26 16:10:08.198018 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.197820 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsm9h\" (UniqueName: \"kubernetes.io/projected/3eb39284-661f-4897-99b5-765bb3aeff16-kube-api-access-xsm9h\") pod \"3eb39284-661f-4897-99b5-765bb3aeff16\" (UID: \"3eb39284-661f-4897-99b5-765bb3aeff16\") " May 26 16:10:08.199214 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.198337 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-bundle" (OuterVolumeSpecName: "bundle") pod "ebd075ef-d961-41b1-8253-85df14a4701b" (UID: "ebd075ef-d961-41b1-8253-85df14a4701b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:10:08.199214 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.198428 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-bundle" (OuterVolumeSpecName: "bundle") pod "7743ce34-6323-4292-8c22-ea9995f53f40" (UID: "7743ce34-6323-4292-8c22-ea9995f53f40"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:10:08.199214 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.198648 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-bundle" (OuterVolumeSpecName: "bundle") pod "3eb39284-661f-4897-99b5-765bb3aeff16" (UID: "3eb39284-661f-4897-99b5-765bb3aeff16"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:10:08.200136 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.200107 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7743ce34-6323-4292-8c22-ea9995f53f40-kube-api-access-8bcwn" (OuterVolumeSpecName: "kube-api-access-8bcwn") pod "7743ce34-6323-4292-8c22-ea9995f53f40" (UID: "7743ce34-6323-4292-8c22-ea9995f53f40"). InnerVolumeSpecName "kube-api-access-8bcwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:10:08.200263 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.200244 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd075ef-d961-41b1-8253-85df14a4701b-kube-api-access-nd9k8" (OuterVolumeSpecName: "kube-api-access-nd9k8") pod "ebd075ef-d961-41b1-8253-85df14a4701b" (UID: "ebd075ef-d961-41b1-8253-85df14a4701b"). InnerVolumeSpecName "kube-api-access-nd9k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:10:08.205725 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.205686 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-util" (OuterVolumeSpecName: "util") pod "ebd075ef-d961-41b1-8253-85df14a4701b" (UID: "ebd075ef-d961-41b1-8253-85df14a4701b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:10:08.206075 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.206055 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb39284-661f-4897-99b5-765bb3aeff16-kube-api-access-xsm9h" (OuterVolumeSpecName: "kube-api-access-xsm9h") pod "3eb39284-661f-4897-99b5-765bb3aeff16" (UID: "3eb39284-661f-4897-99b5-765bb3aeff16"). InnerVolumeSpecName "kube-api-access-xsm9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:10:08.206186 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.206163 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-util" (OuterVolumeSpecName: "util") pod "3eb39284-661f-4897-99b5-765bb3aeff16" (UID: "3eb39284-661f-4897-99b5-765bb3aeff16"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:10:08.207040 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.207022 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-util" (OuterVolumeSpecName: "util") pod "7743ce34-6323-4292-8c22-ea9995f53f40" (UID: "7743ce34-6323-4292-8c22-ea9995f53f40"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:10:08.299204 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.299163 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:08.299204 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.299198 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:08.299204 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.299209 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3eb39284-661f-4897-99b5-765bb3aeff16-util\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:08.299453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.299221 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsm9h\" (UniqueName: \"kubernetes.io/projected/3eb39284-661f-4897-99b5-765bb3aeff16-kube-api-access-xsm9h\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:08.299453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.299235 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bcwn\" (UniqueName: \"kubernetes.io/projected/7743ce34-6323-4292-8c22-ea9995f53f40-kube-api-access-8bcwn\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:08.299453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.299248 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-util\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:08.299453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.299259 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7743ce34-6323-4292-8c22-ea9995f53f40-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:08.299453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.299269 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd075ef-d961-41b1-8253-85df14a4701b-util\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:08.299453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.299281 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd9k8\" (UniqueName: \"kubernetes.io/projected/ebd075ef-d961-41b1-8253-85df14a4701b-kube-api-access-nd9k8\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:08.919625 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.919589 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" May 26 16:10:08.920097 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.919587 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767blkkxc" event={"ID":"3eb39284-661f-4897-99b5-765bb3aeff16","Type":"ContainerDied","Data":"03ab635b88753af11b863f52a994b59a614875f1ab689bd99015159985b8e886"} May 26 16:10:08.920097 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.919700 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ab635b88753af11b863f52a994b59a614875f1ab689bd99015159985b8e886" May 26 16:10:08.921431 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.921411 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" May 26 16:10:08.921536 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.921433 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30dgh2d" event={"ID":"ebd075ef-d961-41b1-8253-85df14a4701b","Type":"ContainerDied","Data":"250bfe5a9df7621e871528762d65960ae25b81fc665687daefc0cb47d8b60d58"} May 26 16:10:08.921536 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.921463 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="250bfe5a9df7621e871528762d65960ae25b81fc665687daefc0cb47d8b60d58" May 26 16:10:08.923230 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.923208 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" May 26 16:10:08.923230 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.923213 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785s9gz" event={"ID":"7743ce34-6323-4292-8c22-ea9995f53f40","Type":"ContainerDied","Data":"62b265380510bd6d8252e17af833b294a89654749afbfc3ecb13222fa5769c34"} May 26 16:10:08.923397 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:08.923246 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62b265380510bd6d8252e17af833b294a89654749afbfc3ecb13222fa5769c34" May 26 16:10:09.039429 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.039404 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:09.207438 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.207291 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbdx7\" (UniqueName: \"kubernetes.io/projected/2a90c77c-62a9-467b-924a-ade0956bec1f-kube-api-access-pbdx7\") pod \"2a90c77c-62a9-467b-924a-ade0956bec1f\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " May 26 16:10:09.207438 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.207433 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-util\") pod \"2a90c77c-62a9-467b-924a-ade0956bec1f\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " May 26 16:10:09.207623 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.207484 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-bundle\") pod \"2a90c77c-62a9-467b-924a-ade0956bec1f\" (UID: \"2a90c77c-62a9-467b-924a-ade0956bec1f\") " May 26 16:10:09.207980 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.207951 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-bundle" (OuterVolumeSpecName: "bundle") pod "2a90c77c-62a9-467b-924a-ade0956bec1f" (UID: "2a90c77c-62a9-467b-924a-ade0956bec1f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:10:09.209371 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.209339 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a90c77c-62a9-467b-924a-ade0956bec1f-kube-api-access-pbdx7" (OuterVolumeSpecName: "kube-api-access-pbdx7") pod "2a90c77c-62a9-467b-924a-ade0956bec1f" (UID: "2a90c77c-62a9-467b-924a-ade0956bec1f"). InnerVolumeSpecName "kube-api-access-pbdx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:10:09.212911 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.212892 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-util" (OuterVolumeSpecName: "util") pod "2a90c77c-62a9-467b-924a-ade0956bec1f" (UID: "2a90c77c-62a9-467b-924a-ade0956bec1f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:10:09.308195 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.308143 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbdx7\" (UniqueName: \"kubernetes.io/projected/2a90c77c-62a9-467b-924a-ade0956bec1f-kube-api-access-pbdx7\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:09.308195 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.308188 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-util\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:09.308195 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.308202 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a90c77c-62a9-467b-924a-ade0956bec1f-bundle\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:10:09.928679 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.928649 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" May 26 16:10:09.929065 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.928648 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88n9mh7" event={"ID":"2a90c77c-62a9-467b-924a-ade0956bec1f","Type":"ContainerDied","Data":"569c3833ee3fbf02e5d55b8d90cf96fe6ea1b5b5b6d8247c0d19b7ce52462c14"} May 26 16:10:09.929065 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:09.928768 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="569c3833ee3fbf02e5d55b8d90cf96fe6ea1b5b5b6d8247c0d19b7ce52462c14" May 26 16:10:24.513795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.513758 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp"] May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514131 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7743ce34-6323-4292-8c22-ea9995f53f40" containerName="util" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514142 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7743ce34-6323-4292-8c22-ea9995f53f40" containerName="util" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514153 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7743ce34-6323-4292-8c22-ea9995f53f40" containerName="extract" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514158 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7743ce34-6323-4292-8c22-ea9995f53f40" containerName="extract" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514165 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd075ef-d961-41b1-8253-85df14a4701b" containerName="extract" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514171 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd075ef-d961-41b1-8253-85df14a4701b" containerName="extract" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514176 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerName="pull" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514181 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerName="pull" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514193 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb39284-661f-4897-99b5-765bb3aeff16" containerName="pull" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514198 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb39284-661f-4897-99b5-765bb3aeff16" containerName="pull" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514205 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerName="extract" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514210 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerName="extract" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514217 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerName="util" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514221 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerName="util" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514229 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb39284-661f-4897-99b5-765bb3aeff16" containerName="util" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514233 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb39284-661f-4897-99b5-765bb3aeff16" containerName="util" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514239 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb39284-661f-4897-99b5-765bb3aeff16" containerName="extract" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514243 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb39284-661f-4897-99b5-765bb3aeff16" containerName="extract" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514253 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd075ef-d961-41b1-8253-85df14a4701b" containerName="pull" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514258 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd075ef-d961-41b1-8253-85df14a4701b" containerName="pull" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514267 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd075ef-d961-41b1-8253-85df14a4701b" containerName="util" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514271 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd075ef-d961-41b1-8253-85df14a4701b" containerName="util" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514277 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7743ce34-6323-4292-8c22-ea9995f53f40" containerName="pull" May 26 16:10:24.514307 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514282 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7743ce34-6323-4292-8c22-ea9995f53f40" containerName="pull" May 26 16:10:24.515092 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514337 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="3eb39284-661f-4897-99b5-765bb3aeff16" containerName="extract" May 26 16:10:24.515092 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514347 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="ebd075ef-d961-41b1-8253-85df14a4701b" containerName="extract" May 26 16:10:24.515092 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514353 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a90c77c-62a9-467b-924a-ade0956bec1f" containerName="extract" May 26 16:10:24.515092 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.514360 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="7743ce34-6323-4292-8c22-ea9995f53f40" containerName="extract" May 26 16:10:24.526289 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.525981 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" May 26 16:10:24.528845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.528801 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" May 26 16:10:24.528975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.528844 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xl5vm\"" May 26 16:10:24.528975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.528877 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" May 26 16:10:24.529542 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.529520 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp"] May 26 16:10:24.529659 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.529641 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" May 26 16:10:24.542145 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.542117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jp5z\" (UniqueName: \"kubernetes.io/projected/b9e67ba5-1cae-4c39-8e2a-4290d13495cb-kube-api-access-4jp5z\") pod \"dns-operator-controller-manager-844548ff4c-2xmqp\" (UID: \"b9e67ba5-1cae-4c39-8e2a-4290d13495cb\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" May 26 16:10:24.643102 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.643063 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4jp5z\" (UniqueName: \"kubernetes.io/projected/b9e67ba5-1cae-4c39-8e2a-4290d13495cb-kube-api-access-4jp5z\") pod \"dns-operator-controller-manager-844548ff4c-2xmqp\" (UID: \"b9e67ba5-1cae-4c39-8e2a-4290d13495cb\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" May 26 16:10:24.654012 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.653985 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jp5z\" (UniqueName: \"kubernetes.io/projected/b9e67ba5-1cae-4c39-8e2a-4290d13495cb-kube-api-access-4jp5z\") pod \"dns-operator-controller-manager-844548ff4c-2xmqp\" (UID: \"b9e67ba5-1cae-4c39-8e2a-4290d13495cb\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" May 26 16:10:24.837318 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.837286 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" May 26 16:10:24.979748 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:24.979606 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp"] May 26 16:10:24.982615 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:10:24.982587 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e67ba5_1cae_4c39_8e2a_4290d13495cb.slice/crio-3c1cac2024914155bf809c3ca5d74a6188fb3c6ea77758a7b783cef518fe10e6 WatchSource:0}: Error finding container 3c1cac2024914155bf809c3ca5d74a6188fb3c6ea77758a7b783cef518fe10e6: Status 404 returned error can't find the container with id 3c1cac2024914155bf809c3ca5d74a6188fb3c6ea77758a7b783cef518fe10e6 May 26 16:10:25.990807 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:25.990760 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" event={"ID":"b9e67ba5-1cae-4c39-8e2a-4290d13495cb","Type":"ContainerStarted","Data":"3c1cac2024914155bf809c3ca5d74a6188fb3c6ea77758a7b783cef518fe10e6"} May 26 16:10:29.297567 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.297535 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs"] May 26 16:10:29.301306 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.301287 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.303794 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.303768 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" May 26 16:10:29.303794 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.303781 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" May 26 16:10:29.304055 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.304039 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7vtq7\"" May 26 16:10:29.310144 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.310125 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs"] May 26 16:10:29.384423 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.384388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.384423 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.384430 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.384644 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.384477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxv4t\" (UniqueName: \"kubernetes.io/projected/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-kube-api-access-kxv4t\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.485223 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.485178 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kxv4t\" (UniqueName: \"kubernetes.io/projected/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-kube-api-access-kxv4t\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.485386 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.485252 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.485386 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.485275 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.485386 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:10:29.485359 2573 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found May 26 16:10:29.485516 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:10:29.485416 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-plugin-serving-cert podName:ec1e07d8-5bdf-4566-a249-24a44c0eb79e nodeName:}" failed. No retries permitted until 2026-05-26 16:10:29.985399751 +0000 UTC m=+578.845322198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-plugin-serving-cert") pod "kuadrant-console-plugin-6c886788f8-vnsbs" (UID: "ec1e07d8-5bdf-4566-a249-24a44c0eb79e") : secret "plugin-serving-cert" not found May 26 16:10:29.485935 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.485913 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.496030 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.496004 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxv4t\" (UniqueName: \"kubernetes.io/projected/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-kube-api-access-kxv4t\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.990367 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.990319 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:29.992709 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:29.992681 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1e07d8-5bdf-4566-a249-24a44c0eb79e-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-vnsbs\" (UID: \"ec1e07d8-5bdf-4566-a249-24a44c0eb79e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:30.009010 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:30.008970 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" event={"ID":"b9e67ba5-1cae-4c39-8e2a-4290d13495cb","Type":"ContainerStarted","Data":"ac087b70bde22bed77ca07953b8b64e2b8b984080cfed84640b30caaac9b452e"} May 26 16:10:30.009149 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:30.009126 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" May 26 16:10:30.036911 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:30.036859 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" podStartSLOduration=2.064130817 podStartE2EDuration="6.036842879s" podCreationTimestamp="2026-05-26 16:10:24 +0000 UTC" firstStartedPulling="2026-05-26 16:10:24.984987728 +0000 UTC m=+573.844910176" lastFinishedPulling="2026-05-26 16:10:28.957699778 +0000 UTC m=+577.817622238" observedRunningTime="2026-05-26 16:10:30.033917822 +0000 UTC m=+578.893840292" watchObservedRunningTime="2026-05-26 16:10:30.036842879 +0000 UTC m=+578.896765339" May 26 16:10:30.211712 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:30.211679 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" May 26 16:10:30.340616 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:30.340587 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs"] May 26 16:10:30.342149 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:10:30.342124 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec1e07d8_5bdf_4566_a249_24a44c0eb79e.slice/crio-642bc01a74c53ca5f8f9fc800c841b27baa6f03fc6c4baf72fdef9dc02d5d104 WatchSource:0}: Error finding container 642bc01a74c53ca5f8f9fc800c841b27baa6f03fc6c4baf72fdef9dc02d5d104: Status 404 returned error can't find the container with id 642bc01a74c53ca5f8f9fc800c841b27baa6f03fc6c4baf72fdef9dc02d5d104 May 26 16:10:31.013702 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:31.013668 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" event={"ID":"ec1e07d8-5bdf-4566-a249-24a44c0eb79e","Type":"ContainerStarted","Data":"642bc01a74c53ca5f8f9fc800c841b27baa6f03fc6c4baf72fdef9dc02d5d104"} May 26 16:10:36.034676 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:36.034641 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" event={"ID":"ec1e07d8-5bdf-4566-a249-24a44c0eb79e","Type":"ContainerStarted","Data":"7acf3362b9b56500d0eff53efc12dd09f73393d59e06ceb0a414d5be43ee913a"} May 26 16:10:36.062375 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:36.062327 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-vnsbs" podStartSLOduration=2.220084196 podStartE2EDuration="7.062313479s" podCreationTimestamp="2026-05-26 16:10:29 +0000 UTC" firstStartedPulling="2026-05-26 16:10:30.343528049 +0000 UTC m=+579.203450498" lastFinishedPulling="2026-05-26 16:10:35.185757319 +0000 UTC m=+584.045679781" observedRunningTime="2026-05-26 16:10:36.060339655 +0000 UTC m=+584.920262136" watchObservedRunningTime="2026-05-26 16:10:36.062313479 +0000 UTC m=+584.922235961" May 26 16:10:41.016110 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:41.016079 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xmqp" May 26 16:10:51.777890 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:51.777863 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:10:51.778334 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:10:51.778201 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:11:14.416171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.416093 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-j5jj5"] May 26 16:11:14.419622 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.419601 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:14.422715 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.422695 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" May 26 16:11:14.428970 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.428946 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-j5jj5"] May 26 16:11:14.459053 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.459019 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-j5jj5"] May 26 16:11:14.497063 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.497028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sptzw\" (UniqueName: \"kubernetes.io/projected/c3d67fbb-b733-4b66-ba79-a3e9aa36e51d-kube-api-access-sptzw\") pod \"limitador-limitador-67566c68b4-j5jj5\" (UID: \"c3d67fbb-b733-4b66-ba79-a3e9aa36e51d\") " pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:14.497211 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.497069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3d67fbb-b733-4b66-ba79-a3e9aa36e51d-config-file\") pod \"limitador-limitador-67566c68b4-j5jj5\" (UID: \"c3d67fbb-b733-4b66-ba79-a3e9aa36e51d\") " pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:14.598485 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.598442 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sptzw\" (UniqueName: \"kubernetes.io/projected/c3d67fbb-b733-4b66-ba79-a3e9aa36e51d-kube-api-access-sptzw\") pod \"limitador-limitador-67566c68b4-j5jj5\" (UID: \"c3d67fbb-b733-4b66-ba79-a3e9aa36e51d\") " pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:14.598485 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.598493 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3d67fbb-b733-4b66-ba79-a3e9aa36e51d-config-file\") pod \"limitador-limitador-67566c68b4-j5jj5\" (UID: \"c3d67fbb-b733-4b66-ba79-a3e9aa36e51d\") " pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:14.599154 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.599134 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3d67fbb-b733-4b66-ba79-a3e9aa36e51d-config-file\") pod \"limitador-limitador-67566c68b4-j5jj5\" (UID: \"c3d67fbb-b733-4b66-ba79-a3e9aa36e51d\") " pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:14.608911 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.608890 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sptzw\" (UniqueName: \"kubernetes.io/projected/c3d67fbb-b733-4b66-ba79-a3e9aa36e51d-kube-api-access-sptzw\") pod \"limitador-limitador-67566c68b4-j5jj5\" (UID: \"c3d67fbb-b733-4b66-ba79-a3e9aa36e51d\") " pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:14.731235 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.731153 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:14.857574 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:14.857550 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-j5jj5"] May 26 16:11:14.859045 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:11:14.859013 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d67fbb_b733_4b66_ba79_a3e9aa36e51d.slice/crio-0c7e7a32b99f1969397f89263707dbdc1aa9c67f2c35a447663d28fc4d37d120 WatchSource:0}: Error finding container 0c7e7a32b99f1969397f89263707dbdc1aa9c67f2c35a447663d28fc4d37d120: Status 404 returned error can't find the container with id 0c7e7a32b99f1969397f89263707dbdc1aa9c67f2c35a447663d28fc4d37d120 May 26 16:11:15.109685 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.109650 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79cbc94b89-cjl4j"] May 26 16:11:15.114421 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.114400 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" May 26 16:11:15.117741 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.117714 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w6sb5\"" May 26 16:11:15.124651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.124625 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-cjl4j"] May 26 16:11:15.188124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.188091 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" event={"ID":"c3d67fbb-b733-4b66-ba79-a3e9aa36e51d","Type":"ContainerStarted","Data":"0c7e7a32b99f1969397f89263707dbdc1aa9c67f2c35a447663d28fc4d37d120"} May 26 16:11:15.203644 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.203622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qd6k\" (UniqueName: \"kubernetes.io/projected/8833b347-7a89-4c11-8442-ca0b4095bfe9-kube-api-access-8qd6k\") pod \"authorino-79cbc94b89-cjl4j\" (UID: \"8833b347-7a89-4c11-8442-ca0b4095bfe9\") " pod="kuadrant-system/authorino-79cbc94b89-cjl4j" May 26 16:11:15.304656 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.304617 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8qd6k\" (UniqueName: \"kubernetes.io/projected/8833b347-7a89-4c11-8442-ca0b4095bfe9-kube-api-access-8qd6k\") pod \"authorino-79cbc94b89-cjl4j\" (UID: \"8833b347-7a89-4c11-8442-ca0b4095bfe9\") " pod="kuadrant-system/authorino-79cbc94b89-cjl4j" May 26 16:11:15.317358 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.317332 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qd6k\" (UniqueName: \"kubernetes.io/projected/8833b347-7a89-4c11-8442-ca0b4095bfe9-kube-api-access-8qd6k\") pod \"authorino-79cbc94b89-cjl4j\" (UID: \"8833b347-7a89-4c11-8442-ca0b4095bfe9\") " pod="kuadrant-system/authorino-79cbc94b89-cjl4j" May 26 16:11:15.424684 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.424600 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" May 26 16:11:15.571282 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:15.571260 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-cjl4j"] May 26 16:11:15.573112 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:11:15.573083 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8833b347_7a89_4c11_8442_ca0b4095bfe9.slice/crio-2fc23fc1b474590134e45cb055e86ee7776ea3587d96792b40cb7b9f38a320b1 WatchSource:0}: Error finding container 2fc23fc1b474590134e45cb055e86ee7776ea3587d96792b40cb7b9f38a320b1: Status 404 returned error can't find the container with id 2fc23fc1b474590134e45cb055e86ee7776ea3587d96792b40cb7b9f38a320b1 May 26 16:11:16.193049 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:16.192959 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" event={"ID":"c3d67fbb-b733-4b66-ba79-a3e9aa36e51d","Type":"ContainerStarted","Data":"7f59cbf6caf5a2590ed3ce2e7275e196daac230166d08bbe8da266c579eb5c93"} May 26 16:11:16.193220 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:16.193068 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:16.194125 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:16.194102 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" event={"ID":"8833b347-7a89-4c11-8442-ca0b4095bfe9","Type":"ContainerStarted","Data":"2fc23fc1b474590134e45cb055e86ee7776ea3587d96792b40cb7b9f38a320b1"} May 26 16:11:16.216026 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:16.215984 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" podStartSLOduration=1.1606376919999999 podStartE2EDuration="2.215970725s" podCreationTimestamp="2026-05-26 16:11:14 +0000 UTC" firstStartedPulling="2026-05-26 16:11:14.860892967 +0000 UTC m=+623.720815414" lastFinishedPulling="2026-05-26 16:11:15.916225998 +0000 UTC m=+624.776148447" observedRunningTime="2026-05-26 16:11:16.215972094 +0000 UTC m=+625.075894564" watchObservedRunningTime="2026-05-26 16:11:16.215970725 +0000 UTC m=+625.075893192" May 26 16:11:18.205943 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:18.205897 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" event={"ID":"8833b347-7a89-4c11-8442-ca0b4095bfe9","Type":"ContainerStarted","Data":"a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6"} May 26 16:11:18.253514 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:18.253459 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" podStartSLOduration=0.750419866 podStartE2EDuration="3.253443822s" podCreationTimestamp="2026-05-26 16:11:15 +0000 UTC" firstStartedPulling="2026-05-26 16:11:15.574538784 +0000 UTC m=+624.434461231" lastFinishedPulling="2026-05-26 16:11:18.07756274 +0000 UTC m=+626.937485187" observedRunningTime="2026-05-26 16:11:18.252533181 +0000 UTC m=+627.112455650" watchObservedRunningTime="2026-05-26 16:11:18.253443822 +0000 UTC m=+627.113366290" May 26 16:11:27.202884 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:27.202851 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-67566c68b4-j5jj5" May 26 16:11:39.801069 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:39.801031 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-cjl4j"] May 26 16:11:39.803323 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:39.801243 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" podUID="8833b347-7a89-4c11-8442-ca0b4095bfe9" containerName="authorino" containerID="cri-o://a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6" gracePeriod=30 May 26 16:11:40.053380 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.053308 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" May 26 16:11:40.225525 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.225493 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qd6k\" (UniqueName: \"kubernetes.io/projected/8833b347-7a89-4c11-8442-ca0b4095bfe9-kube-api-access-8qd6k\") pod \"8833b347-7a89-4c11-8442-ca0b4095bfe9\" (UID: \"8833b347-7a89-4c11-8442-ca0b4095bfe9\") " May 26 16:11:40.227643 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.227609 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8833b347-7a89-4c11-8442-ca0b4095bfe9-kube-api-access-8qd6k" (OuterVolumeSpecName: "kube-api-access-8qd6k") pod "8833b347-7a89-4c11-8442-ca0b4095bfe9" (UID: "8833b347-7a89-4c11-8442-ca0b4095bfe9"). InnerVolumeSpecName "kube-api-access-8qd6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:11:40.291024 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.290992 2573 generic.go:358] "Generic (PLEG): container finished" podID="8833b347-7a89-4c11-8442-ca0b4095bfe9" containerID="a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6" exitCode=0 May 26 16:11:40.291189 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.291044 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" May 26 16:11:40.291189 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.291076 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" event={"ID":"8833b347-7a89-4c11-8442-ca0b4095bfe9","Type":"ContainerDied","Data":"a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6"} May 26 16:11:40.291189 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.291113 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-cjl4j" event={"ID":"8833b347-7a89-4c11-8442-ca0b4095bfe9","Type":"ContainerDied","Data":"2fc23fc1b474590134e45cb055e86ee7776ea3587d96792b40cb7b9f38a320b1"} May 26 16:11:40.291189 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.291129 2573 scope.go:117] "RemoveContainer" containerID="a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6" May 26 16:11:40.300381 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.300364 2573 scope.go:117] "RemoveContainer" containerID="a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6" May 26 16:11:40.300605 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:11:40.300588 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6\": container with ID starting with a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6 not found: ID does not exist" containerID="a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6" May 26 16:11:40.300646 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.300614 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6"} err="failed to get container status \"a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6\": rpc error: code = NotFound desc = could not find container \"a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6\": container with ID starting with a6d7e2d571bfc3fb1f9b9523d0fb3e8affa697313c44fdfa994f56406a9178f6 not found: ID does not exist" May 26 16:11:40.321978 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.321900 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-cjl4j"] May 26 16:11:40.326746 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.326712 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qd6k\" (UniqueName: \"kubernetes.io/projected/8833b347-7a89-4c11-8442-ca0b4095bfe9-kube-api-access-8qd6k\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:11:40.329518 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:40.329492 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-cjl4j"] May 26 16:11:41.874324 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:11:41.874291 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8833b347-7a89-4c11-8442-ca0b4095bfe9" path="/var/lib/kubelet/pods/8833b347-7a89-4c11-8442-ca0b4095bfe9/volumes" May 26 16:13:32.883855 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.883805 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-spwx2"] May 26 16:13:32.884261 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.884197 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8833b347-7a89-4c11-8442-ca0b4095bfe9" containerName="authorino" May 26 16:13:32.884261 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.884209 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8833b347-7a89-4c11-8442-ca0b4095bfe9" containerName="authorino" May 26 16:13:32.884335 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.884288 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="8833b347-7a89-4c11-8442-ca0b4095bfe9" containerName="authorino" May 26 16:13:32.887348 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.887331 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-spwx2" May 26 16:13:32.890057 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.890025 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8x2dr\"" May 26 16:13:32.890188 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.890063 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" May 26 16:13:32.890188 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.890114 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" May 26 16:13:32.890188 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.890069 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" May 26 16:13:32.895696 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:32.895677 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-spwx2"] May 26 16:13:33.009472 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:33.009422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzj9\" (UniqueName: \"kubernetes.io/projected/cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6-kube-api-access-8hzj9\") pod \"s3-init-spwx2\" (UID: \"cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6\") " pod="kserve/s3-init-spwx2" May 26 16:13:33.110546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:33.110510 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzj9\" (UniqueName: \"kubernetes.io/projected/cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6-kube-api-access-8hzj9\") pod \"s3-init-spwx2\" (UID: \"cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6\") " pod="kserve/s3-init-spwx2" May 26 16:13:33.120427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:33.120396 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzj9\" (UniqueName: \"kubernetes.io/projected/cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6-kube-api-access-8hzj9\") pod \"s3-init-spwx2\" (UID: \"cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6\") " pod="kserve/s3-init-spwx2" May 26 16:13:33.197208 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:33.197123 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-spwx2" May 26 16:13:33.327197 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:33.327167 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-spwx2"] May 26 16:13:33.328448 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:13:33.328421 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd6a3428_c45c_46e4_9d1f_b10d1c74a4b6.slice/crio-463b3cbb9708e0c64893e7d17e087acea778f2028fa5f979d53e2208fd92ba71 WatchSource:0}: Error finding container 463b3cbb9708e0c64893e7d17e087acea778f2028fa5f979d53e2208fd92ba71: Status 404 returned error can't find the container with id 463b3cbb9708e0c64893e7d17e087acea778f2028fa5f979d53e2208fd92ba71 May 26 16:13:33.330217 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:33.330200 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:13:33.745751 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:33.745713 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-spwx2" event={"ID":"cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6","Type":"ContainerStarted","Data":"463b3cbb9708e0c64893e7d17e087acea778f2028fa5f979d53e2208fd92ba71"} May 26 16:13:38.770162 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:38.770122 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-spwx2" event={"ID":"cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6","Type":"ContainerStarted","Data":"55a97de3e87c243d87925a1b7689af47ede132dfeedc5e5ccba3278d4b8fb865"} May 26 16:13:38.787292 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:38.787242 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-spwx2" podStartSLOduration=2.293284187 podStartE2EDuration="6.78722686s" podCreationTimestamp="2026-05-26 16:13:32 +0000 UTC" firstStartedPulling="2026-05-26 16:13:33.330322233 +0000 UTC m=+762.190244691" lastFinishedPulling="2026-05-26 16:13:37.824264913 +0000 UTC m=+766.684187364" observedRunningTime="2026-05-26 16:13:38.786226693 +0000 UTC m=+767.646149163" watchObservedRunningTime="2026-05-26 16:13:38.78722686 +0000 UTC m=+767.647149328" May 26 16:13:41.783275 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:41.783241 2573 generic.go:358] "Generic (PLEG): container finished" podID="cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6" containerID="55a97de3e87c243d87925a1b7689af47ede132dfeedc5e5ccba3278d4b8fb865" exitCode=0 May 26 16:13:41.783667 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:41.783314 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-spwx2" event={"ID":"cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6","Type":"ContainerDied","Data":"55a97de3e87c243d87925a1b7689af47ede132dfeedc5e5ccba3278d4b8fb865"} May 26 16:13:42.922137 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:42.922113 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-spwx2" May 26 16:13:43.008022 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:43.007992 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hzj9\" (UniqueName: \"kubernetes.io/projected/cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6-kube-api-access-8hzj9\") pod \"cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6\" (UID: \"cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6\") " May 26 16:13:43.010194 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:43.010160 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6-kube-api-access-8hzj9" (OuterVolumeSpecName: "kube-api-access-8hzj9") pod "cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6" (UID: "cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6"). InnerVolumeSpecName "kube-api-access-8hzj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:13:43.108875 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:43.108750 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hzj9\" (UniqueName: \"kubernetes.io/projected/cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6-kube-api-access-8hzj9\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:13:43.792302 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:43.792270 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-spwx2" May 26 16:13:43.792476 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:43.792301 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-spwx2" event={"ID":"cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6","Type":"ContainerDied","Data":"463b3cbb9708e0c64893e7d17e087acea778f2028fa5f979d53e2208fd92ba71"} May 26 16:13:43.792476 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:13:43.792332 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="463b3cbb9708e0c64893e7d17e087acea778f2028fa5f979d53e2208fd92ba71" May 26 16:14:10.083138 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.083105 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs"] May 26 16:14:10.083597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.083536 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6" containerName="s3-init" May 26 16:14:10.083597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.083547 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6" containerName="s3-init" May 26 16:14:10.083672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.083627 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6" containerName="s3-init" May 26 16:14:10.111135 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.111108 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs"] May 26 16:14:10.111291 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.111239 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.114007 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.113983 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" May 26 16:14:10.114958 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.114932 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" May 26 16:14:10.115101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.114939 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\"" May 26 16:14:10.115101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.114956 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\"" May 26 16:14:10.115222 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.114956 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-998w7\"" May 26 16:14:10.260090 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.260056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdcb95d-145e-4e14-9ef4-5d962c021437-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.260090 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.260094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwr7q\" (UniqueName: \"kubernetes.io/projected/0fdcb95d-145e-4e14-9ef4-5d962c021437-kube-api-access-fwr7q\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.260387 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.260134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.260387 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.260218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.260387 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.260269 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.260387 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.260293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.361722 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.361638 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdcb95d-145e-4e14-9ef4-5d962c021437-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.361722 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.361672 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fwr7q\" (UniqueName: \"kubernetes.io/projected/0fdcb95d-145e-4e14-9ef4-5d962c021437-kube-api-access-fwr7q\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.361949 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.361722 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.361949 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.361766 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.361949 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.361801 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.361949 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.361826 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.362254 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.362231 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.362254 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.362247 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.362350 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.362286 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.362350 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.362311 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.364257 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.364240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdcb95d-145e-4e14-9ef4-5d962c021437-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.370372 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.370342 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwr7q\" (UniqueName: \"kubernetes.io/projected/0fdcb95d-145e-4e14-9ef4-5d962c021437-kube-api-access-fwr7q\") pod \"scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.421738 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.421703 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:10.558226 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.558199 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs"] May 26 16:14:10.560225 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:14:10.560200 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdcb95d_145e_4e14_9ef4_5d962c021437.slice/crio-41e45885a7fb466714d82d36a04b86913cea9411ed5d76579dbad1e59c768d1f WatchSource:0}: Error finding container 41e45885a7fb466714d82d36a04b86913cea9411ed5d76579dbad1e59c768d1f: Status 404 returned error can't find the container with id 41e45885a7fb466714d82d36a04b86913cea9411ed5d76579dbad1e59c768d1f May 26 16:14:10.899492 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:10.899450 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" event={"ID":"0fdcb95d-145e-4e14-9ef4-5d962c021437","Type":"ContainerStarted","Data":"41e45885a7fb466714d82d36a04b86913cea9411ed5d76579dbad1e59c768d1f"} May 26 16:14:13.913927 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:13.913893 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" event={"ID":"0fdcb95d-145e-4e14-9ef4-5d962c021437","Type":"ContainerStarted","Data":"7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f"} May 26 16:14:14.919398 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:14.919365 2573 generic.go:358] "Generic (PLEG): container finished" podID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerID="7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f" exitCode=0 May 26 16:14:14.919898 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:14.919426 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" event={"ID":"0fdcb95d-145e-4e14-9ef4-5d962c021437","Type":"ContainerDied","Data":"7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f"} May 26 16:14:16.930461 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:16.930387 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" event={"ID":"0fdcb95d-145e-4e14-9ef4-5d962c021437","Type":"ContainerStarted","Data":"4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506"} May 26 16:14:47.072918 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:47.072877 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" event={"ID":"0fdcb95d-145e-4e14-9ef4-5d962c021437","Type":"ContainerStarted","Data":"d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262"} May 26 16:14:47.073394 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:47.073136 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:47.077095 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:47.077072 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:47.100309 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:47.100267 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" podStartSLOduration=1.090370651 podStartE2EDuration="37.100253803s" podCreationTimestamp="2026-05-26 16:14:10 +0000 UTC" firstStartedPulling="2026-05-26 16:14:10.562183577 +0000 UTC m=+799.422106024" lastFinishedPulling="2026-05-26 16:14:46.572066715 +0000 UTC m=+835.431989176" observedRunningTime="2026-05-26 16:14:47.097024564 +0000 UTC m=+835.956947032" watchObservedRunningTime="2026-05-26 16:14:47.100253803 +0000 UTC m=+835.960176273" May 26 16:14:50.422309 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:50.422268 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:14:50.422690 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:14:50.422337 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:15:00.424292 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:00.424262 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:15:00.425524 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:00.425502 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:15:36.916159 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:36.916123 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs"] May 26 16:15:36.916704 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:36.916455 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="main" containerID="cri-o://4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506" gracePeriod=30 May 26 16:15:36.916704 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:36.916497 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="tokenizer" containerID="cri-o://d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262" gracePeriod=30 May 26 16:15:37.074036 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:15:37.074005 2573 logging.go:55] [core] [Channel #47 SubChannel #48]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.48:9003", ServerName: "10.133.0.48:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.48:9003: connect: connection refused" May 26 16:15:37.269809 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:37.269777 2573 generic.go:358] "Generic (PLEG): container finished" podID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerID="4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506" exitCode=0 May 26 16:15:37.270014 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:37.269876 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" event={"ID":"0fdcb95d-145e-4e14-9ef4-5d962c021437","Type":"ContainerDied","Data":"4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506"} May 26 16:15:38.074640 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.074597 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.48:9003\" within 1s: context deadline exceeded" May 26 16:15:38.167971 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.167946 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:15:38.275199 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.275166 2573 generic.go:358] "Generic (PLEG): container finished" podID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerID="d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262" exitCode=0 May 26 16:15:38.275386 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.275235 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" May 26 16:15:38.275386 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.275251 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" event={"ID":"0fdcb95d-145e-4e14-9ef4-5d962c021437","Type":"ContainerDied","Data":"d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262"} May 26 16:15:38.275386 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.275296 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs" event={"ID":"0fdcb95d-145e-4e14-9ef4-5d962c021437","Type":"ContainerDied","Data":"41e45885a7fb466714d82d36a04b86913cea9411ed5d76579dbad1e59c768d1f"} May 26 16:15:38.275386 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.275313 2573 scope.go:117] "RemoveContainer" containerID="d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262" May 26 16:15:38.284234 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.284217 2573 scope.go:117] "RemoveContainer" containerID="4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506" May 26 16:15:38.291384 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291359 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-cache\") pod \"0fdcb95d-145e-4e14-9ef4-5d962c021437\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " May 26 16:15:38.291487 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291395 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwr7q\" (UniqueName: \"kubernetes.io/projected/0fdcb95d-145e-4e14-9ef4-5d962c021437-kube-api-access-fwr7q\") pod \"0fdcb95d-145e-4e14-9ef4-5d962c021437\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " May 26 16:15:38.291487 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291426 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-kserve-provision-location\") pod \"0fdcb95d-145e-4e14-9ef4-5d962c021437\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " May 26 16:15:38.291566 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291489 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-uds\") pod \"0fdcb95d-145e-4e14-9ef4-5d962c021437\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " May 26 16:15:38.291566 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291536 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-tmp\") pod \"0fdcb95d-145e-4e14-9ef4-5d962c021437\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " May 26 16:15:38.291663 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291577 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdcb95d-145e-4e14-9ef4-5d962c021437-tls-certs\") pod \"0fdcb95d-145e-4e14-9ef4-5d962c021437\" (UID: \"0fdcb95d-145e-4e14-9ef4-5d962c021437\") " May 26 16:15:38.291720 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291661 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "0fdcb95d-145e-4e14-9ef4-5d962c021437" (UID: "0fdcb95d-145e-4e14-9ef4-5d962c021437"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:15:38.291814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291789 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0fdcb95d-145e-4e14-9ef4-5d962c021437" (UID: "0fdcb95d-145e-4e14-9ef4-5d962c021437"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:15:38.291916 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291897 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "0fdcb95d-145e-4e14-9ef4-5d962c021437" (UID: "0fdcb95d-145e-4e14-9ef4-5d962c021437"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:15:38.291962 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291919 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:15:38.291962 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.291939 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:15:38.292282 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.292257 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0fdcb95d-145e-4e14-9ef4-5d962c021437" (UID: "0fdcb95d-145e-4e14-9ef4-5d962c021437"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:15:38.292579 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.292559 2573 scope.go:117] "RemoveContainer" containerID="7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f" May 26 16:15:38.293904 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.293882 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdcb95d-145e-4e14-9ef4-5d962c021437-kube-api-access-fwr7q" (OuterVolumeSpecName: "kube-api-access-fwr7q") pod "0fdcb95d-145e-4e14-9ef4-5d962c021437" (UID: "0fdcb95d-145e-4e14-9ef4-5d962c021437"). InnerVolumeSpecName "kube-api-access-fwr7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:15:38.294001 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.293917 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdcb95d-145e-4e14-9ef4-5d962c021437-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fdcb95d-145e-4e14-9ef4-5d962c021437" (UID: "0fdcb95d-145e-4e14-9ef4-5d962c021437"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:15:38.311583 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.311561 2573 scope.go:117] "RemoveContainer" containerID="d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262" May 26 16:15:38.311904 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:15:38.311883 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262\": container with ID starting with d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262 not found: ID does not exist" containerID="d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262" May 26 16:15:38.311974 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.311916 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262"} err="failed to get container status \"d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262\": rpc error: code = NotFound desc = could not find container \"d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262\": container with ID starting with d037dba3ce020f25ec0cdbe3f62742be296591fd7859575f93a8b8be0ae70262 not found: ID does not exist" May 26 16:15:38.311974 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.311937 2573 scope.go:117] "RemoveContainer" containerID="4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506" May 26 16:15:38.312192 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:15:38.312170 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506\": container with ID starting with 4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506 not found: ID does not exist" containerID="4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506" May 26 16:15:38.312255 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.312204 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506"} err="failed to get container status \"4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506\": rpc error: code = NotFound desc = could not find container \"4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506\": container with ID starting with 4a6efb7828f00c56667e2c127380dd0072b00ffa01ef7853ee064c4c4fc6d506 not found: ID does not exist" May 26 16:15:38.312255 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.312229 2573 scope.go:117] "RemoveContainer" containerID="7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f" May 26 16:15:38.312468 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:15:38.312450 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f\": container with ID starting with 7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f not found: ID does not exist" containerID="7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f" May 26 16:15:38.312510 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.312474 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f"} err="failed to get container status \"7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f\": rpc error: code = NotFound desc = could not find container \"7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f\": container with ID starting with 7ef799ee4c6ec333dbb09bc504786a67a21381fb5cf5f6ddd0fa0a2a7f3a468f not found: ID does not exist" May 26 16:15:38.392651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.392609 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:15:38.392651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.392646 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdcb95d-145e-4e14-9ef4-5d962c021437-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:15:38.392651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.392656 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwr7q\" (UniqueName: \"kubernetes.io/projected/0fdcb95d-145e-4e14-9ef4-5d962c021437-kube-api-access-fwr7q\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:15:38.392909 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.392667 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fdcb95d-145e-4e14-9ef4-5d962c021437-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:15:38.601341 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.601311 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs"] May 26 16:15:38.607254 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:38.607229 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-84d9dbd5vs"] May 26 16:15:39.874320 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:39.874286 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" path="/var/lib/kubelet/pods/0fdcb95d-145e-4e14-9ef4-5d962c021437/volumes" May 26 16:15:44.314606 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.314576 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv"] May 26 16:15:44.315020 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.315006 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="storage-initializer" May 26 16:15:44.315064 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.315023 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="storage-initializer" May 26 16:15:44.315064 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.315035 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="main" May 26 16:15:44.315064 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.315042 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="main" May 26 16:15:44.315153 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.315067 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="tokenizer" May 26 16:15:44.315153 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.315073 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="tokenizer" May 26 16:15:44.315153 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.315134 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="main" May 26 16:15:44.315153 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.315142 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="0fdcb95d-145e-4e14-9ef4-5d962c021437" containerName="tokenizer" May 26 16:15:44.318919 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.318897 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.321411 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.321389 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" May 26 16:15:44.321546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.321390 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\"" May 26 16:15:44.322165 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.322144 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" May 26 16:15:44.322284 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.322153 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\"" May 26 16:15:44.322284 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.322196 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-z52j9\"" May 26 16:15:44.331041 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.331016 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv"] May 26 16:15:44.340276 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.340250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.340424 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.340287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.340424 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.340324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.340424 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.340350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgj9t\" (UniqueName: \"kubernetes.io/projected/551e4041-d3f0-4130-858c-e110fa714f3e-kube-api-access-jgj9t\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.340600 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.340422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.340600 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.340493 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551e4041-d3f0-4130-858c-e110fa714f3e-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.441982 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.441949 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551e4041-d3f0-4130-858c-e110fa714f3e-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.442184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.442020 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.442184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.442048 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.442184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.442076 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.442184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.442095 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgj9t\" (UniqueName: \"kubernetes.io/projected/551e4041-d3f0-4130-858c-e110fa714f3e-kube-api-access-jgj9t\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.442184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.442132 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.442543 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.442520 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.442698 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.442538 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.442698 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.442593 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.442698 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.442615 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.444520 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.444495 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551e4041-d3f0-4130-858c-e110fa714f3e-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.451464 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.451436 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgj9t\" (UniqueName: \"kubernetes.io/projected/551e4041-d3f0-4130-858c-e110fa714f3e-kube-api-access-jgj9t\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.630716 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.630619 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:44.769343 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:44.769317 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv"] May 26 16:15:44.771404 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:15:44.771369 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551e4041_d3f0_4130_858c_e110fa714f3e.slice/crio-670cc71e57c3fb306ec88b038c91b8c371733f61e57abb16beae264f0dc920b8 WatchSource:0}: Error finding container 670cc71e57c3fb306ec88b038c91b8c371733f61e57abb16beae264f0dc920b8: Status 404 returned error can't find the container with id 670cc71e57c3fb306ec88b038c91b8c371733f61e57abb16beae264f0dc920b8 May 26 16:15:45.306308 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:45.306269 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" event={"ID":"551e4041-d3f0-4130-858c-e110fa714f3e","Type":"ContainerStarted","Data":"3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62"} May 26 16:15:45.306308 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:45.306307 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" event={"ID":"551e4041-d3f0-4130-858c-e110fa714f3e","Type":"ContainerStarted","Data":"670cc71e57c3fb306ec88b038c91b8c371733f61e57abb16beae264f0dc920b8"} May 26 16:15:46.311153 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:46.311119 2573 generic.go:358] "Generic (PLEG): container finished" podID="551e4041-d3f0-4130-858c-e110fa714f3e" containerID="3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62" exitCode=0 May 26 16:15:46.311581 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:46.311179 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" event={"ID":"551e4041-d3f0-4130-858c-e110fa714f3e","Type":"ContainerDied","Data":"3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62"} May 26 16:15:47.317545 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:47.317509 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" event={"ID":"551e4041-d3f0-4130-858c-e110fa714f3e","Type":"ContainerStarted","Data":"63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2"} May 26 16:15:47.317545 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:47.317546 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" event={"ID":"551e4041-d3f0-4130-858c-e110fa714f3e","Type":"ContainerStarted","Data":"41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a"} May 26 16:15:47.317999 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:47.317635 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:47.343924 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:47.343873 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" podStartSLOduration=3.343856579 podStartE2EDuration="3.343856579s" podCreationTimestamp="2026-05-26 16:15:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:15:47.34191324 +0000 UTC m=+896.201835711" watchObservedRunningTime="2026-05-26 16:15:47.343856579 +0000 UTC m=+896.203779049" May 26 16:15:51.814622 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:51.814585 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:15:51.815782 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:51.815758 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:15:54.631650 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:54.631607 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:54.631650 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:54.631655 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:54.634287 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:54.634263 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:15:55.351434 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:15:55.351396 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:16:16.355881 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:16.355825 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:16:32.927474 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.927432 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm"] May 26 16:16:32.931458 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.931439 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:32.933686 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.933668 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-bvf7z\"" May 26 16:16:32.933953 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.933933 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\"" May 26 16:16:32.945528 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.945504 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm"] May 26 16:16:32.977822 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.977786 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:32.978002 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.977825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:32.978002 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.977868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:32.978002 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.977921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npng\" (UniqueName: \"kubernetes.io/projected/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kube-api-access-4npng\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:32.978002 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.977951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:32.978165 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:32.978018 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.078657 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.078620 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.078901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.078674 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.078901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.078695 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.078901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.078713 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.078901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.078769 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4npng\" (UniqueName: \"kubernetes.io/projected/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kube-api-access-4npng\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.078901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.078818 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.079189 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.079112 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.079189 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.079163 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.079259 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.079203 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.079259 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.079240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.081174 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.081157 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.086908 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.086890 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npng\" (UniqueName: \"kubernetes.io/projected/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kube-api-access-4npng\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.241155 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.241066 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:33.378486 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.378459 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm"] May 26 16:16:33.379750 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:16:33.379724 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b7daec_5a2f_4a90_8e29_7ef7c29dcb08.slice/crio-44f086791b5c4028b0845574e97882ab0b180fb66fed3cb823bd0c8a4328e27a WatchSource:0}: Error finding container 44f086791b5c4028b0845574e97882ab0b180fb66fed3cb823bd0c8a4328e27a: Status 404 returned error can't find the container with id 44f086791b5c4028b0845574e97882ab0b180fb66fed3cb823bd0c8a4328e27a May 26 16:16:33.503555 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.503459 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" event={"ID":"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08","Type":"ContainerStarted","Data":"447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8"} May 26 16:16:33.503555 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:33.503507 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" event={"ID":"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08","Type":"ContainerStarted","Data":"44f086791b5c4028b0845574e97882ab0b180fb66fed3cb823bd0c8a4328e27a"} May 26 16:16:34.509473 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:34.509373 2573 generic.go:358] "Generic (PLEG): container finished" podID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerID="447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8" exitCode=0 May 26 16:16:34.509473 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:34.509410 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" event={"ID":"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08","Type":"ContainerDied","Data":"447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8"} May 26 16:16:34.521372 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:34.521345 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv"] May 26 16:16:34.521658 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:34.521619 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="main" containerID="cri-o://41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a" gracePeriod=30 May 26 16:16:34.521729 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:34.521651 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="tokenizer" containerID="cri-o://63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2" gracePeriod=30 May 26 16:16:35.350814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.350773 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="tokenizer" probeResult="failure" output="Get \"http://10.133.0.49:8082/healthz\": dial tcp 10.133.0.49:8082: connect: connection refused" May 26 16:16:35.515848 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.515800 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" event={"ID":"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08","Type":"ContainerStarted","Data":"60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e"} May 26 16:16:35.516341 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.515865 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" event={"ID":"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08","Type":"ContainerStarted","Data":"d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220"} May 26 16:16:35.516341 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.516027 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:35.517941 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.517918 2573 generic.go:358] "Generic (PLEG): container finished" podID="551e4041-d3f0-4130-858c-e110fa714f3e" containerID="41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a" exitCode=0 May 26 16:16:35.518073 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.517976 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" event={"ID":"551e4041-d3f0-4130-858c-e110fa714f3e","Type":"ContainerDied","Data":"41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a"} May 26 16:16:35.555887 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.555823 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" podStartSLOduration=3.555808387 podStartE2EDuration="3.555808387s" podCreationTimestamp="2026-05-26 16:16:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:16:35.553438092 +0000 UTC m=+944.413360562" watchObservedRunningTime="2026-05-26 16:16:35.555808387 +0000 UTC m=+944.415730855" May 26 16:16:35.787681 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.787655 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:16:35.803043 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803018 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgj9t\" (UniqueName: \"kubernetes.io/projected/551e4041-d3f0-4130-858c-e110fa714f3e-kube-api-access-jgj9t\") pod \"551e4041-d3f0-4130-858c-e110fa714f3e\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " May 26 16:16:35.803188 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803067 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-uds\") pod \"551e4041-d3f0-4130-858c-e110fa714f3e\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " May 26 16:16:35.803188 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803099 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-tmp\") pod \"551e4041-d3f0-4130-858c-e110fa714f3e\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " May 26 16:16:35.803188 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803121 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-cache\") pod \"551e4041-d3f0-4130-858c-e110fa714f3e\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " May 26 16:16:35.803188 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803166 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551e4041-d3f0-4130-858c-e110fa714f3e-tls-certs\") pod \"551e4041-d3f0-4130-858c-e110fa714f3e\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " May 26 16:16:35.803404 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803233 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-kserve-provision-location\") pod \"551e4041-d3f0-4130-858c-e110fa714f3e\" (UID: \"551e4041-d3f0-4130-858c-e110fa714f3e\") " May 26 16:16:35.803404 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803365 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "551e4041-d3f0-4130-858c-e110fa714f3e" (UID: "551e4041-d3f0-4130-858c-e110fa714f3e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:16:35.803521 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803458 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "551e4041-d3f0-4130-858c-e110fa714f3e" (UID: "551e4041-d3f0-4130-858c-e110fa714f3e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:16:35.803521 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803470 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "551e4041-d3f0-4130-858c-e110fa714f3e" (UID: "551e4041-d3f0-4130-858c-e110fa714f3e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:16:35.803628 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803581 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:16:35.803628 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803600 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:16:35.803628 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.803614 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:16:35.804167 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.804135 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "551e4041-d3f0-4130-858c-e110fa714f3e" (UID: "551e4041-d3f0-4130-858c-e110fa714f3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:16:35.805204 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.805175 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551e4041-d3f0-4130-858c-e110fa714f3e-kube-api-access-jgj9t" (OuterVolumeSpecName: "kube-api-access-jgj9t") pod "551e4041-d3f0-4130-858c-e110fa714f3e" (UID: "551e4041-d3f0-4130-858c-e110fa714f3e"). InnerVolumeSpecName "kube-api-access-jgj9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:16:35.805408 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.805387 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551e4041-d3f0-4130-858c-e110fa714f3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "551e4041-d3f0-4130-858c-e110fa714f3e" (UID: "551e4041-d3f0-4130-858c-e110fa714f3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:16:35.905229 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.905172 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551e4041-d3f0-4130-858c-e110fa714f3e-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:16:35.905229 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.905223 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgj9t\" (UniqueName: \"kubernetes.io/projected/551e4041-d3f0-4130-858c-e110fa714f3e-kube-api-access-jgj9t\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:16:35.905431 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:35.905239 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551e4041-d3f0-4130-858c-e110fa714f3e-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:16:36.523911 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.523875 2573 generic.go:358] "Generic (PLEG): container finished" podID="551e4041-d3f0-4130-858c-e110fa714f3e" containerID="63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2" exitCode=0 May 26 16:16:36.524368 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.523960 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" event={"ID":"551e4041-d3f0-4130-858c-e110fa714f3e","Type":"ContainerDied","Data":"63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2"} May 26 16:16:36.524368 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.523972 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" May 26 16:16:36.524368 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.524008 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv" event={"ID":"551e4041-d3f0-4130-858c-e110fa714f3e","Type":"ContainerDied","Data":"670cc71e57c3fb306ec88b038c91b8c371733f61e57abb16beae264f0dc920b8"} May 26 16:16:36.524368 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.524031 2573 scope.go:117] "RemoveContainer" containerID="63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2" May 26 16:16:36.532999 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.532983 2573 scope.go:117] "RemoveContainer" containerID="41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a" May 26 16:16:36.540628 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.540609 2573 scope.go:117] "RemoveContainer" containerID="3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62" May 26 16:16:36.546057 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.546033 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv"] May 26 16:16:36.548523 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.548496 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-6cc4cdxkfv"] May 26 16:16:36.549943 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.549929 2573 scope.go:117] "RemoveContainer" containerID="63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2" May 26 16:16:36.550192 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:16:36.550172 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2\": container with ID starting with 63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2 not found: ID does not exist" containerID="63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2" May 26 16:16:36.550271 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.550200 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2"} err="failed to get container status \"63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2\": rpc error: code = NotFound desc = could not find container \"63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2\": container with ID starting with 63d7ffebadf0422cfe7f91aea8a7e4e86f1301c8217d1f453db332b33d7befd2 not found: ID does not exist" May 26 16:16:36.550271 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.550218 2573 scope.go:117] "RemoveContainer" containerID="41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a" May 26 16:16:36.550432 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:16:36.550414 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a\": container with ID starting with 41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a not found: ID does not exist" containerID="41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a" May 26 16:16:36.550498 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.550436 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a"} err="failed to get container status \"41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a\": rpc error: code = NotFound desc = could not find container \"41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a\": container with ID starting with 41cf86b5d38a0704f10f5efb3d47340588e102c261914c83eb5605170f51360a not found: ID does not exist" May 26 16:16:36.550498 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.550451 2573 scope.go:117] "RemoveContainer" containerID="3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62" May 26 16:16:36.550631 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:16:36.550615 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62\": container with ID starting with 3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62 not found: ID does not exist" containerID="3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62" May 26 16:16:36.550698 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:36.550632 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62"} err="failed to get container status \"3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62\": rpc error: code = NotFound desc = could not find container \"3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62\": container with ID starting with 3383dff2c1ec29b33a6a2d992341278240663f6cd10a5ce7b953a60178a81c62 not found: ID does not exist" May 26 16:16:37.874996 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:37.874961 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" path="/var/lib/kubelet/pods/551e4041-d3f0-4130-858c-e110fa714f3e/volumes" May 26 16:16:41.481845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.481799 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw"] May 26 16:16:41.484212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.482459 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="tokenizer" May 26 16:16:41.484212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.482476 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="tokenizer" May 26 16:16:41.484212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.482509 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="storage-initializer" May 26 16:16:41.484212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.482540 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="storage-initializer" May 26 16:16:41.484212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.482573 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="main" May 26 16:16:41.484212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.482583 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="main" May 26 16:16:41.484212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.482680 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="tokenizer" May 26 16:16:41.484212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.482696 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="551e4041-d3f0-4130-858c-e110fa714f3e" containerName="main" May 26 16:16:41.486462 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.486443 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.488902 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.488878 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-54c75\"" May 26 16:16:41.488902 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.488898 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\"" May 26 16:16:41.501668 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.501646 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw"] May 26 16:16:41.556880 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.556820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.557073 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.556903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.557073 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.556927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.557073 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.556977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtxw\" (UniqueName: \"kubernetes.io/projected/f2e37e30-846e-4286-9088-b95d37293b89-kube-api-access-nrtxw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.557073 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.557026 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.557233 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.557086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e37e30-846e-4286-9088-b95d37293b89-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.658552 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.658519 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e37e30-846e-4286-9088-b95d37293b89-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.658727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.658559 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.658727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.658601 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.658727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.658620 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.658727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.658662 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtxw\" (UniqueName: \"kubernetes.io/projected/f2e37e30-846e-4286-9088-b95d37293b89-kube-api-access-nrtxw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.658727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.658723 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.659110 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.659078 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.659213 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.659111 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.659213 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.659189 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.659312 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.659216 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.661019 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.661001 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e37e30-846e-4286-9088-b95d37293b89-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.667167 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.667142 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtxw\" (UniqueName: \"kubernetes.io/projected/f2e37e30-846e-4286-9088-b95d37293b89-kube-api-access-nrtxw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.796952 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.796918 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:41.940322 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:41.940289 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw"] May 26 16:16:41.942100 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:16:41.942065 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e37e30_846e_4286_9088_b95d37293b89.slice/crio-478896eaab510037cedb6df7b86e9066048261c21b1f5d0bd08306738940bd6a WatchSource:0}: Error finding container 478896eaab510037cedb6df7b86e9066048261c21b1f5d0bd08306738940bd6a: Status 404 returned error can't find the container with id 478896eaab510037cedb6df7b86e9066048261c21b1f5d0bd08306738940bd6a May 26 16:16:42.551803 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:42.551769 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" event={"ID":"f2e37e30-846e-4286-9088-b95d37293b89","Type":"ContainerStarted","Data":"de96c7f92aeb39034aa60b28180a1836db101fd5e7f488db5ada3098def9cca8"} May 26 16:16:42.551803 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:42.551805 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" event={"ID":"f2e37e30-846e-4286-9088-b95d37293b89","Type":"ContainerStarted","Data":"478896eaab510037cedb6df7b86e9066048261c21b1f5d0bd08306738940bd6a"} May 26 16:16:43.242169 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:43.242119 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:43.242169 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:43.242181 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:43.245131 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:43.245096 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:43.556978 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:43.556939 2573 generic.go:358] "Generic (PLEG): container finished" podID="f2e37e30-846e-4286-9088-b95d37293b89" containerID="de96c7f92aeb39034aa60b28180a1836db101fd5e7f488db5ada3098def9cca8" exitCode=0 May 26 16:16:43.557486 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:43.557026 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" event={"ID":"f2e37e30-846e-4286-9088-b95d37293b89","Type":"ContainerDied","Data":"de96c7f92aeb39034aa60b28180a1836db101fd5e7f488db5ada3098def9cca8"} May 26 16:16:43.558670 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:43.558649 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:16:44.562750 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:44.562707 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" event={"ID":"f2e37e30-846e-4286-9088-b95d37293b89","Type":"ContainerStarted","Data":"19e4808b1dd9a196543c3c1e8dfea5356ce9e55340e5ddd8dad10259cb27e1b2"} May 26 16:16:44.562750 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:44.562757 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" event={"ID":"f2e37e30-846e-4286-9088-b95d37293b89","Type":"ContainerStarted","Data":"e43a80569214cf5bdb4216904b89cee81ac513d7a1912a914bb335ee4f36f88f"} May 26 16:16:44.563354 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:44.563066 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:44.584302 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:44.584246 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" podStartSLOduration=3.584225655 podStartE2EDuration="3.584225655s" podCreationTimestamp="2026-05-26 16:16:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:16:44.583119817 +0000 UTC m=+953.443042307" watchObservedRunningTime="2026-05-26 16:16:44.584225655 +0000 UTC m=+953.444148126" May 26 16:16:51.797136 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:51.797103 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:51.797136 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:51.797142 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:51.799990 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:51.799965 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:16:52.596793 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:16:52.596760 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:17:04.564507 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:17:04.564475 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:17:13.601458 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:17:13.601375 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:20:51.850680 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:20:51.850650 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:20:51.857201 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:20:51.857177 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:20:59.590487 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:20:59.590451 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm"] May 26 16:20:59.590975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:20:59.590759 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="main" containerID="cri-o://d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220" gracePeriod=30 May 26 16:20:59.591054 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:20:59.590953 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="tokenizer" containerID="cri-o://60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e" gracePeriod=30 May 26 16:21:00.631451 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.631415 2573 generic.go:358] "Generic (PLEG): container finished" podID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerID="d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220" exitCode=0 May 26 16:21:00.631913 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.631465 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" event={"ID":"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08","Type":"ContainerDied","Data":"d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220"} May 26 16:21:00.846125 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.846101 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:21:00.927731 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.927634 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kserve-provision-location\") pod \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " May 26 16:21:00.927731 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.927688 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-uds\") pod \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " May 26 16:21:00.927731 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.927729 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-cache\") pod \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " May 26 16:21:00.928051 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.927802 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tls-certs\") pod \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " May 26 16:21:00.928051 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.927824 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4npng\" (UniqueName: \"kubernetes.io/projected/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kube-api-access-4npng\") pod \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " May 26 16:21:00.928051 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.927881 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-tmp\") pod \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\" (UID: \"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08\") " May 26 16:21:00.928051 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.928027 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" (UID: "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:21:00.928051 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.928042 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" (UID: "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:21:00.928270 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.928185 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:21:00.928270 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.928203 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:21:00.928384 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.928265 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" (UID: "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:21:00.928494 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.928472 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" (UID: "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:21:00.930544 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.930525 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kube-api-access-4npng" (OuterVolumeSpecName: "kube-api-access-4npng") pod "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" (UID: "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08"). InnerVolumeSpecName "kube-api-access-4npng". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:21:00.930602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:00.930527 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" (UID: "e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:21:01.029309 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.029271 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:21:01.029309 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.029304 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:21:01.029309 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.029318 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4npng\" (UniqueName: \"kubernetes.io/projected/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-kube-api-access-4npng\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:21:01.029541 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.029330 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:21:01.637523 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.637492 2573 generic.go:358] "Generic (PLEG): container finished" podID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerID="60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e" exitCode=0 May 26 16:21:01.638003 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.637572 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" May 26 16:21:01.638003 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.637607 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" event={"ID":"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08","Type":"ContainerDied","Data":"60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e"} May 26 16:21:01.638003 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.637634 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm" event={"ID":"e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08","Type":"ContainerDied","Data":"44f086791b5c4028b0845574e97882ab0b180fb66fed3cb823bd0c8a4328e27a"} May 26 16:21:01.638003 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.637658 2573 scope.go:117] "RemoveContainer" containerID="60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e" May 26 16:21:01.647574 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.647557 2573 scope.go:117] "RemoveContainer" containerID="d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220" May 26 16:21:01.655486 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.655470 2573 scope.go:117] "RemoveContainer" containerID="447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8" May 26 16:21:01.662558 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.662532 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm"] May 26 16:21:01.665319 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.665298 2573 scope.go:117] "RemoveContainer" containerID="60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e" May 26 16:21:01.665657 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:21:01.665633 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e\": container with ID starting with 60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e not found: ID does not exist" containerID="60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e" May 26 16:21:01.665725 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.665670 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e"} err="failed to get container status \"60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e\": rpc error: code = NotFound desc = could not find container \"60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e\": container with ID starting with 60632d6be59524cc02ae5972089c428a231d95520490658c165aa0ee8ba6c61e not found: ID does not exist" May 26 16:21:01.665725 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.665699 2573 scope.go:117] "RemoveContainer" containerID="d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220" May 26 16:21:01.666015 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:21:01.665997 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220\": container with ID starting with d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220 not found: ID does not exist" containerID="d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220" May 26 16:21:01.666084 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.666021 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220"} err="failed to get container status \"d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220\": rpc error: code = NotFound desc = could not find container \"d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220\": container with ID starting with d4245bec500d32f18d015af68922d6263483b026b6cd70417d50cb2a46fea220 not found: ID does not exist" May 26 16:21:01.666084 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.666039 2573 scope.go:117] "RemoveContainer" containerID="447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8" May 26 16:21:01.666277 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:21:01.666262 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8\": container with ID starting with 447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8 not found: ID does not exist" containerID="447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8" May 26 16:21:01.666322 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.666281 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8"} err="failed to get container status \"447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8\": rpc error: code = NotFound desc = could not find container \"447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8\": container with ID starting with 447242eaf5bda415731caf6cd9d8b22281ce733d4b94c06d3e8726c582a7bbf8 not found: ID does not exist" May 26 16:21:01.667237 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.667220 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche5chzm"] May 26 16:21:01.876621 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:01.876586 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" path="/var/lib/kubelet/pods/e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08/volumes" May 26 16:21:25.857638 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.857600 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f"] May 26 16:21:25.858124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.858083 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="main" May 26 16:21:25.858124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.858098 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="main" May 26 16:21:25.858124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.858111 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="tokenizer" May 26 16:21:25.858124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.858116 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="tokenizer" May 26 16:21:25.858124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.858126 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="storage-initializer" May 26 16:21:25.858288 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.858131 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="storage-initializer" May 26 16:21:25.858288 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.858201 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="main" May 26 16:21:25.858288 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.858213 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0b7daec-5a2f-4a90-8e29-7ef7c29dcb08" containerName="tokenizer" May 26 16:21:25.861516 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.861500 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:25.863688 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.863663 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-b89m6\"" May 26 16:21:25.863818 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.863668 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\"" May 26 16:21:25.874898 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.874873 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f"] May 26 16:21:25.965317 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.965275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:25.965499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.965330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:25.965499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.965402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10863224-11d5-4c89-af13-c14d659a49b5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:25.965499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.965448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:25.965499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.965476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8sh\" (UniqueName: \"kubernetes.io/projected/10863224-11d5-4c89-af13-c14d659a49b5-kube-api-access-7x8sh\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:25.965499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:25.965496 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066241 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066204 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066449 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066250 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10863224-11d5-4c89-af13-c14d659a49b5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066449 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066285 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066449 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066322 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8sh\" (UniqueName: \"kubernetes.io/projected/10863224-11d5-4c89-af13-c14d659a49b5-kube-api-access-7x8sh\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066449 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066351 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066740 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066450 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066740 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066688 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066740 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066702 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066740 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066738 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.066937 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.066813 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.068935 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.068908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10863224-11d5-4c89-af13-c14d659a49b5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.073618 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.073596 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8sh\" (UniqueName: \"kubernetes.io/projected/10863224-11d5-4c89-af13-c14d659a49b5-kube-api-access-7x8sh\") pod \"custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.172624 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.172521 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:26.311132 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.311105 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f"] May 26 16:21:26.312857 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:21:26.312805 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10863224_11d5_4c89_af13_c14d659a49b5.slice/crio-70cd240752ba180abfc8fa66a3d4d2fb0f38e1d17366fc116f824127f5d00709 WatchSource:0}: Error finding container 70cd240752ba180abfc8fa66a3d4d2fb0f38e1d17366fc116f824127f5d00709: Status 404 returned error can't find the container with id 70cd240752ba180abfc8fa66a3d4d2fb0f38e1d17366fc116f824127f5d00709 May 26 16:21:26.314943 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.314924 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:21:26.744924 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.744880 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" event={"ID":"10863224-11d5-4c89-af13-c14d659a49b5","Type":"ContainerStarted","Data":"fc2bb9f7de759c930a92a7daeab0af37f3813491f34dc892e55a0a5edc9a92b6"} May 26 16:21:26.745121 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:26.744936 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" event={"ID":"10863224-11d5-4c89-af13-c14d659a49b5","Type":"ContainerStarted","Data":"70cd240752ba180abfc8fa66a3d4d2fb0f38e1d17366fc116f824127f5d00709"} May 26 16:21:27.749759 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:27.749718 2573 generic.go:358] "Generic (PLEG): container finished" podID="10863224-11d5-4c89-af13-c14d659a49b5" containerID="fc2bb9f7de759c930a92a7daeab0af37f3813491f34dc892e55a0a5edc9a92b6" exitCode=0 May 26 16:21:27.750183 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:27.749799 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" event={"ID":"10863224-11d5-4c89-af13-c14d659a49b5","Type":"ContainerDied","Data":"fc2bb9f7de759c930a92a7daeab0af37f3813491f34dc892e55a0a5edc9a92b6"} May 26 16:21:28.755609 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:28.755532 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" event={"ID":"10863224-11d5-4c89-af13-c14d659a49b5","Type":"ContainerStarted","Data":"f6305c999c3817182943682bbfe4c5d6ea80c10c752ec627f024d7045f5d5291"} May 26 16:21:28.755609 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:28.755614 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" event={"ID":"10863224-11d5-4c89-af13-c14d659a49b5","Type":"ContainerStarted","Data":"87f5cb697faf86d25c225be423e563fbb27601291f8be19af4e71fe9e31e1baf"} May 26 16:21:28.756116 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:28.755721 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:28.778477 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:28.778432 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" podStartSLOduration=3.778415677 podStartE2EDuration="3.778415677s" podCreationTimestamp="2026-05-26 16:21:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:21:28.777075439 +0000 UTC m=+1237.636997908" watchObservedRunningTime="2026-05-26 16:21:28.778415677 +0000 UTC m=+1237.638338143" May 26 16:21:36.173496 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:36.173456 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:36.174008 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:36.173511 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:36.176217 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:36.176194 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:36.788004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:36.787970 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:21:57.792975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:21:57.792941 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:23:22.237133 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:22.237039 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f"] May 26 16:23:22.237620 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:22.237472 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="main" containerID="cri-o://87f5cb697faf86d25c225be423e563fbb27601291f8be19af4e71fe9e31e1baf" gracePeriod=30 May 26 16:23:22.237620 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:22.237519 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="tokenizer" containerID="cri-o://f6305c999c3817182943682bbfe4c5d6ea80c10c752ec627f024d7045f5d5291" gracePeriod=30 May 26 16:23:23.208321 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.208291 2573 generic.go:358] "Generic (PLEG): container finished" podID="10863224-11d5-4c89-af13-c14d659a49b5" containerID="f6305c999c3817182943682bbfe4c5d6ea80c10c752ec627f024d7045f5d5291" exitCode=0 May 26 16:23:23.208321 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.208317 2573 generic.go:358] "Generic (PLEG): container finished" podID="10863224-11d5-4c89-af13-c14d659a49b5" containerID="87f5cb697faf86d25c225be423e563fbb27601291f8be19af4e71fe9e31e1baf" exitCode=0 May 26 16:23:23.208505 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.208351 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" event={"ID":"10863224-11d5-4c89-af13-c14d659a49b5","Type":"ContainerDied","Data":"f6305c999c3817182943682bbfe4c5d6ea80c10c752ec627f024d7045f5d5291"} May 26 16:23:23.208505 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.208376 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" event={"ID":"10863224-11d5-4c89-af13-c14d659a49b5","Type":"ContainerDied","Data":"87f5cb697faf86d25c225be423e563fbb27601291f8be19af4e71fe9e31e1baf"} May 26 16:23:23.309369 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.309346 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:23:23.408287 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408213 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10863224-11d5-4c89-af13-c14d659a49b5-tls-certs\") pod \"10863224-11d5-4c89-af13-c14d659a49b5\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " May 26 16:23:23.408287 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408283 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x8sh\" (UniqueName: \"kubernetes.io/projected/10863224-11d5-4c89-af13-c14d659a49b5-kube-api-access-7x8sh\") pod \"10863224-11d5-4c89-af13-c14d659a49b5\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " May 26 16:23:23.408445 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408307 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-cache\") pod \"10863224-11d5-4c89-af13-c14d659a49b5\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " May 26 16:23:23.408445 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408335 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-uds\") pod \"10863224-11d5-4c89-af13-c14d659a49b5\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " May 26 16:23:23.408445 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408352 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-kserve-provision-location\") pod \"10863224-11d5-4c89-af13-c14d659a49b5\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " May 26 16:23:23.408445 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408404 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-tmp\") pod \"10863224-11d5-4c89-af13-c14d659a49b5\" (UID: \"10863224-11d5-4c89-af13-c14d659a49b5\") " May 26 16:23:23.408658 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408634 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "10863224-11d5-4c89-af13-c14d659a49b5" (UID: "10863224-11d5-4c89-af13-c14d659a49b5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:23:23.408712 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408635 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "10863224-11d5-4c89-af13-c14d659a49b5" (UID: "10863224-11d5-4c89-af13-c14d659a49b5"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:23:23.408756 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408742 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "10863224-11d5-4c89-af13-c14d659a49b5" (UID: "10863224-11d5-4c89-af13-c14d659a49b5"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:23:23.408790 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408769 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:23:23.408790 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.408781 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:23:23.409269 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.409239 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10863224-11d5-4c89-af13-c14d659a49b5" (UID: "10863224-11d5-4c89-af13-c14d659a49b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:23:23.410471 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.410451 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10863224-11d5-4c89-af13-c14d659a49b5-kube-api-access-7x8sh" (OuterVolumeSpecName: "kube-api-access-7x8sh") pod "10863224-11d5-4c89-af13-c14d659a49b5" (UID: "10863224-11d5-4c89-af13-c14d659a49b5"). InnerVolumeSpecName "kube-api-access-7x8sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:23:23.410556 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.410536 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10863224-11d5-4c89-af13-c14d659a49b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10863224-11d5-4c89-af13-c14d659a49b5" (UID: "10863224-11d5-4c89-af13-c14d659a49b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:23:23.509878 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.509820 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x8sh\" (UniqueName: \"kubernetes.io/projected/10863224-11d5-4c89-af13-c14d659a49b5-kube-api-access-7x8sh\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:23:23.509878 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.509874 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:23:23.510029 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.509891 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/10863224-11d5-4c89-af13-c14d659a49b5-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:23:23.510029 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:23.509904 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10863224-11d5-4c89-af13-c14d659a49b5-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:23:24.214387 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:24.214293 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" May 26 16:23:24.214387 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:24.214301 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f" event={"ID":"10863224-11d5-4c89-af13-c14d659a49b5","Type":"ContainerDied","Data":"70cd240752ba180abfc8fa66a3d4d2fb0f38e1d17366fc116f824127f5d00709"} May 26 16:23:24.214387 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:24.214355 2573 scope.go:117] "RemoveContainer" containerID="f6305c999c3817182943682bbfe4c5d6ea80c10c752ec627f024d7045f5d5291" May 26 16:23:24.226197 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:24.226137 2573 scope.go:117] "RemoveContainer" containerID="87f5cb697faf86d25c225be423e563fbb27601291f8be19af4e71fe9e31e1baf" May 26 16:23:24.233477 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:24.233452 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f"] May 26 16:23:24.235227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:24.235204 2573 scope.go:117] "RemoveContainer" containerID="fc2bb9f7de759c930a92a7daeab0af37f3813491f34dc892e55a0a5edc9a92b6" May 26 16:23:24.237308 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:24.237285 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6f9b8f4fk4f8f"] May 26 16:23:25.874722 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:25.874678 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10863224-11d5-4c89-af13-c14d659a49b5" path="/var/lib/kubelet/pods/10863224-11d5-4c89-af13-c14d659a49b5/volumes" May 26 16:23:40.677349 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.677312 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx"] May 26 16:23:40.677767 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.677742 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="tokenizer" May 26 16:23:40.677767 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.677755 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="tokenizer" May 26 16:23:40.677861 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.677776 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="storage-initializer" May 26 16:23:40.677861 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.677782 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="storage-initializer" May 26 16:23:40.677861 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.677791 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="main" May 26 16:23:40.677861 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.677800 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="main" May 26 16:23:40.677990 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.677877 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="main" May 26 16:23:40.677990 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.677890 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="10863224-11d5-4c89-af13-c14d659a49b5" containerName="tokenizer" May 26 16:23:40.682930 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.682910 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.685207 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.685187 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-wbbjz\"" May 26 16:23:40.685323 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.685187 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\"" May 26 16:23:40.689861 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.689815 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx"] May 26 16:23:40.761242 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.761208 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.761427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.761265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.761427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.761284 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.761427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.761302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.761427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.761324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2bd\" (UniqueName: \"kubernetes.io/projected/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kube-api-access-8d2bd\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.761427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.761347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.862512 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.862477 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.862512 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.862515 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.862775 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.862540 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.862775 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.862560 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2bd\" (UniqueName: \"kubernetes.io/projected/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kube-api-access-8d2bd\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.862775 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.862579 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.862775 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.862641 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.863024 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.862966 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.863024 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.863005 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.863106 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.863029 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.863106 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.863067 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.865074 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.865056 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.870166 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.870147 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2bd\" (UniqueName: \"kubernetes.io/projected/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kube-api-access-8d2bd\") pod \"router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:40.994134 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:40.994021 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:41.126963 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:41.126935 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx"] May 26 16:23:41.128392 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:23:41.128367 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5f3991_ea93_4c87_aa1e_d3cd80c756f1.slice/crio-1843dc20192e5ce34027f1cd5a9b0c776c27f2f172266465fefbc96a03ab393a WatchSource:0}: Error finding container 1843dc20192e5ce34027f1cd5a9b0c776c27f2f172266465fefbc96a03ab393a: Status 404 returned error can't find the container with id 1843dc20192e5ce34027f1cd5a9b0c776c27f2f172266465fefbc96a03ab393a May 26 16:23:41.288815 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:41.288779 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" event={"ID":"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1","Type":"ContainerStarted","Data":"e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66"} May 26 16:23:41.288815 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:41.288818 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" event={"ID":"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1","Type":"ContainerStarted","Data":"1843dc20192e5ce34027f1cd5a9b0c776c27f2f172266465fefbc96a03ab393a"} May 26 16:23:42.294375 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:42.294341 2573 generic.go:358] "Generic (PLEG): container finished" podID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerID="e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66" exitCode=0 May 26 16:23:42.294723 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:42.294391 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" event={"ID":"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1","Type":"ContainerDied","Data":"e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66"} May 26 16:23:43.301154 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:43.301110 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" event={"ID":"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1","Type":"ContainerStarted","Data":"c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a"} May 26 16:23:43.301536 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:43.301166 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" event={"ID":"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1","Type":"ContainerStarted","Data":"ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c"} May 26 16:23:43.301536 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:43.301192 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:43.325344 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:43.325296 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" podStartSLOduration=3.325281855 podStartE2EDuration="3.325281855s" podCreationTimestamp="2026-05-26 16:23:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:23:43.323614423 +0000 UTC m=+1372.183536892" watchObservedRunningTime="2026-05-26 16:23:43.325281855 +0000 UTC m=+1372.185204323" May 26 16:23:50.994853 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:50.994794 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:50.995324 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:50.994949 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:50.997776 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:50.997749 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:23:51.335246 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:23:51.335219 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:24:13.345205 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:24:13.345175 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:25:51.892227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:25:51.892198 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:25:51.903456 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:25:51.903430 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:27:56.687004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:56.686972 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx"] May 26 16:27:56.687602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:56.687302 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="main" containerID="cri-o://ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c" gracePeriod=30 May 26 16:27:56.687602 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:56.687437 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="tokenizer" containerID="cri-o://c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a" gracePeriod=30 May 26 16:27:57.332017 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:57.331983 2573 generic.go:358] "Generic (PLEG): container finished" podID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerID="ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c" exitCode=0 May 26 16:27:57.332225 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:57.332056 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" event={"ID":"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1","Type":"ContainerDied","Data":"ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c"} May 26 16:27:57.943621 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:57.943594 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:27:58.059933 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.059900 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-tmp\") pod \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " May 26 16:27:58.060152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.059983 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d2bd\" (UniqueName: \"kubernetes.io/projected/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kube-api-access-8d2bd\") pod \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " May 26 16:27:58.060152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060041 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tls-certs\") pod \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " May 26 16:27:58.060152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060073 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kserve-provision-location\") pod \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " May 26 16:27:58.060152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060104 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-uds\") pod \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " May 26 16:27:58.060152 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060142 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-cache\") pod \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\" (UID: \"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1\") " May 26 16:27:58.060416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060276 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" (UID: "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:27:58.060416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060334 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" (UID: "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:27:58.060519 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060441 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:27:58.060519 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060459 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:27:58.060519 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060457 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" (UID: "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:27:58.060979 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.060952 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" (UID: "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:27:58.062156 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.062128 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kube-api-access-8d2bd" (OuterVolumeSpecName: "kube-api-access-8d2bd") pod "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" (UID: "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1"). InnerVolumeSpecName "kube-api-access-8d2bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:27:58.062270 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.062198 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" (UID: "ab5f3991-ea93-4c87-aa1e-d3cd80c756f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:27:58.161200 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.161158 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2bd\" (UniqueName: \"kubernetes.io/projected/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kube-api-access-8d2bd\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:27:58.161200 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.161192 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:27:58.161200 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.161206 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:27:58.161450 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.161215 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:27:58.337418 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.337329 2573 generic.go:358] "Generic (PLEG): container finished" podID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerID="c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a" exitCode=0 May 26 16:27:58.337418 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.337383 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" event={"ID":"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1","Type":"ContainerDied","Data":"c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a"} May 26 16:27:58.337418 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.337403 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" May 26 16:27:58.337418 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.337419 2573 scope.go:117] "RemoveContainer" containerID="c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a" May 26 16:27:58.337749 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.337409 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx" event={"ID":"ab5f3991-ea93-4c87-aa1e-d3cd80c756f1","Type":"ContainerDied","Data":"1843dc20192e5ce34027f1cd5a9b0c776c27f2f172266465fefbc96a03ab393a"} May 26 16:27:58.347457 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.347437 2573 scope.go:117] "RemoveContainer" containerID="ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c" May 26 16:27:58.355802 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.355780 2573 scope.go:117] "RemoveContainer" containerID="e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66" May 26 16:27:58.361037 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.361005 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx"] May 26 16:27:58.365172 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.365142 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7cd8874bc4-m52gx"] May 26 16:27:58.366865 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.366824 2573 scope.go:117] "RemoveContainer" containerID="c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a" May 26 16:27:58.367189 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:27:58.367170 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a\": container with ID starting with c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a not found: ID does not exist" containerID="c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a" May 26 16:27:58.367259 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.367204 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a"} err="failed to get container status \"c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a\": rpc error: code = NotFound desc = could not find container \"c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a\": container with ID starting with c14ea4510a3e2ef4fc598906b833eb3c84c1cd673ae08efaeda14bd5656bbb0a not found: ID does not exist" May 26 16:27:58.367259 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.367226 2573 scope.go:117] "RemoveContainer" containerID="ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c" May 26 16:27:58.367542 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:27:58.367524 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c\": container with ID starting with ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c not found: ID does not exist" containerID="ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c" May 26 16:27:58.367607 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.367547 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c"} err="failed to get container status \"ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c\": rpc error: code = NotFound desc = could not find container \"ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c\": container with ID starting with ff703b9adb8b263d8e6a6a743af8dbaefa0f7171c3e9edacd7327ec9a278890c not found: ID does not exist" May 26 16:27:58.367607 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.367567 2573 scope.go:117] "RemoveContainer" containerID="e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66" May 26 16:27:58.367843 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:27:58.367812 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66\": container with ID starting with e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66 not found: ID does not exist" containerID="e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66" May 26 16:27:58.367916 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:58.367885 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66"} err="failed to get container status \"e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66\": rpc error: code = NotFound desc = could not find container \"e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66\": container with ID starting with e5a7d7b6415438d794431bc9e3fcb40d6dcb56b4759a9ba66b9bff7704af6b66 not found: ID does not exist" May 26 16:27:59.880445 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:27:59.876093 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" path="/var/lib/kubelet/pods/ab5f3991-ea93-4c87-aa1e-d3cd80c756f1/volumes" May 26 16:28:05.060768 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.060731 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m"] May 26 16:28:05.061179 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.061127 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="main" May 26 16:28:05.061179 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.061139 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="main" May 26 16:28:05.061179 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.061157 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="tokenizer" May 26 16:28:05.061179 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.061162 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="tokenizer" May 26 16:28:05.061179 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.061179 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="storage-initializer" May 26 16:28:05.061339 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.061186 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="storage-initializer" May 26 16:28:05.061339 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.061247 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="tokenizer" May 26 16:28:05.061339 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.061259 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="ab5f3991-ea93-4c87-aa1e-d3cd80c756f1" containerName="main" May 26 16:28:05.066285 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.066260 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.074479 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.074450 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\"" May 26 16:28:05.074621 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.074523 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-7kbbp\"" May 26 16:28:05.084665 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.084630 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m"] May 26 16:28:05.116416 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.116381 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.116573 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.116427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsvxt\" (UniqueName: \"kubernetes.io/projected/78f56c9d-8fd4-44a1-991d-6377b51823cc-kube-api-access-zsvxt\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.116573 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.116546 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f56c9d-8fd4-44a1-991d-6377b51823cc-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.116573 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.116568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.116680 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.116602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.116680 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.116645 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.217521 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.217485 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f56c9d-8fd4-44a1-991d-6377b51823cc-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.217521 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.217520 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.217772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.217546 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.217772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.217568 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.217772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.217611 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.217772 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.217635 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zsvxt\" (UniqueName: \"kubernetes.io/projected/78f56c9d-8fd4-44a1-991d-6377b51823cc-kube-api-access-zsvxt\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.218023 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.217972 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.218023 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.218005 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.218142 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.218029 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.218142 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.218092 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.220231 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.220207 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f56c9d-8fd4-44a1-991d-6377b51823cc-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.231392 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.231359 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsvxt\" (UniqueName: \"kubernetes.io/projected/78f56c9d-8fd4-44a1-991d-6377b51823cc-kube-api-access-zsvxt\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.376768 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.376675 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:05.514133 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.514105 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m"] May 26 16:28:05.514883 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:28:05.514857 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f56c9d_8fd4_44a1_991d_6377b51823cc.slice/crio-8b1cb3032efaffd5eefb917afa568df2253b84388de9ffa57d5ce820a9b369d8 WatchSource:0}: Error finding container 8b1cb3032efaffd5eefb917afa568df2253b84388de9ffa57d5ce820a9b369d8: Status 404 returned error can't find the container with id 8b1cb3032efaffd5eefb917afa568df2253b84388de9ffa57d5ce820a9b369d8 May 26 16:28:05.516927 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:05.516908 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:28:06.374245 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:06.374207 2573 generic.go:358] "Generic (PLEG): container finished" podID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerID="79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9" exitCode=0 May 26 16:28:06.374650 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:06.374289 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" event={"ID":"78f56c9d-8fd4-44a1-991d-6377b51823cc","Type":"ContainerDied","Data":"79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9"} May 26 16:28:06.374650 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:06.374323 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" event={"ID":"78f56c9d-8fd4-44a1-991d-6377b51823cc","Type":"ContainerStarted","Data":"8b1cb3032efaffd5eefb917afa568df2253b84388de9ffa57d5ce820a9b369d8"} May 26 16:28:07.380872 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:07.380815 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" event={"ID":"78f56c9d-8fd4-44a1-991d-6377b51823cc","Type":"ContainerStarted","Data":"a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538"} May 26 16:28:07.380872 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:07.380865 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" event={"ID":"78f56c9d-8fd4-44a1-991d-6377b51823cc","Type":"ContainerStarted","Data":"1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499"} May 26 16:28:07.381452 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:07.380988 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:07.402187 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:07.402127 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" podStartSLOduration=2.402108424 podStartE2EDuration="2.402108424s" podCreationTimestamp="2026-05-26 16:28:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:28:07.401234909 +0000 UTC m=+1636.261157376" watchObservedRunningTime="2026-05-26 16:28:07.402108424 +0000 UTC m=+1636.262030897" May 26 16:28:15.377003 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:15.376955 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:15.377414 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:15.377171 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:15.379796 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:15.379770 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:15.418467 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:15.418445 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:28:37.426723 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:28:37.426691 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:30:51.929117 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:30:51.929089 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:30:51.939427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:30:51.939405 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:31:02.925341 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:02.925297 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m"] May 26 16:31:02.925901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:02.925750 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="main" containerID="cri-o://1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499" gracePeriod=30 May 26 16:31:02.925901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:02.925792 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="tokenizer" containerID="cri-o://a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538" gracePeriod=30 May 26 16:31:03.124242 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:03.124207 2573 generic.go:358] "Generic (PLEG): container finished" podID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerID="1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499" exitCode=0 May 26 16:31:03.124415 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:03.124283 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" event={"ID":"78f56c9d-8fd4-44a1-991d-6377b51823cc","Type":"ContainerDied","Data":"1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499"} May 26 16:31:04.291112 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.291051 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:31:04.392332 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392293 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsvxt\" (UniqueName: \"kubernetes.io/projected/78f56c9d-8fd4-44a1-991d-6377b51823cc-kube-api-access-zsvxt\") pod \"78f56c9d-8fd4-44a1-991d-6377b51823cc\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " May 26 16:31:04.392546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392357 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-cache\") pod \"78f56c9d-8fd4-44a1-991d-6377b51823cc\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " May 26 16:31:04.392546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392376 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-kserve-provision-location\") pod \"78f56c9d-8fd4-44a1-991d-6377b51823cc\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " May 26 16:31:04.392546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392437 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f56c9d-8fd4-44a1-991d-6377b51823cc-tls-certs\") pod \"78f56c9d-8fd4-44a1-991d-6377b51823cc\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " May 26 16:31:04.392546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392468 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-uds\") pod \"78f56c9d-8fd4-44a1-991d-6377b51823cc\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " May 26 16:31:04.392546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392523 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-tmp\") pod \"78f56c9d-8fd4-44a1-991d-6377b51823cc\" (UID: \"78f56c9d-8fd4-44a1-991d-6377b51823cc\") " May 26 16:31:04.392813 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392634 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "78f56c9d-8fd4-44a1-991d-6377b51823cc" (UID: "78f56c9d-8fd4-44a1-991d-6377b51823cc"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:31:04.392813 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392782 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "78f56c9d-8fd4-44a1-991d-6377b51823cc" (UID: "78f56c9d-8fd4-44a1-991d-6377b51823cc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:31:04.392934 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392913 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "78f56c9d-8fd4-44a1-991d-6377b51823cc" (UID: "78f56c9d-8fd4-44a1-991d-6377b51823cc"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:31:04.392977 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392941 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:04.392977 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.392962 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:04.393426 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.393397 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78f56c9d-8fd4-44a1-991d-6377b51823cc" (UID: "78f56c9d-8fd4-44a1-991d-6377b51823cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:31:04.394552 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.394521 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f56c9d-8fd4-44a1-991d-6377b51823cc-kube-api-access-zsvxt" (OuterVolumeSpecName: "kube-api-access-zsvxt") pod "78f56c9d-8fd4-44a1-991d-6377b51823cc" (UID: "78f56c9d-8fd4-44a1-991d-6377b51823cc"). InnerVolumeSpecName "kube-api-access-zsvxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:31:04.394659 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.394555 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f56c9d-8fd4-44a1-991d-6377b51823cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78f56c9d-8fd4-44a1-991d-6377b51823cc" (UID: "78f56c9d-8fd4-44a1-991d-6377b51823cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:31:04.493782 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.493735 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:04.493782 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.493777 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f56c9d-8fd4-44a1-991d-6377b51823cc-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:04.493782 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.493788 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f56c9d-8fd4-44a1-991d-6377b51823cc-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:04.493782 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:04.493800 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsvxt\" (UniqueName: \"kubernetes.io/projected/78f56c9d-8fd4-44a1-991d-6377b51823cc-kube-api-access-zsvxt\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:05.134465 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.134426 2573 generic.go:358] "Generic (PLEG): container finished" podID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerID="a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538" exitCode=0 May 26 16:31:05.134669 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.134468 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" event={"ID":"78f56c9d-8fd4-44a1-991d-6377b51823cc","Type":"ContainerDied","Data":"a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538"} May 26 16:31:05.134669 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.134508 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" May 26 16:31:05.134669 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.134517 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m" event={"ID":"78f56c9d-8fd4-44a1-991d-6377b51823cc","Type":"ContainerDied","Data":"8b1cb3032efaffd5eefb917afa568df2253b84388de9ffa57d5ce820a9b369d8"} May 26 16:31:05.134669 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.134537 2573 scope.go:117] "RemoveContainer" containerID="a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538" May 26 16:31:05.144686 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.144531 2573 scope.go:117] "RemoveContainer" containerID="1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499" May 26 16:31:05.152792 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.152773 2573 scope.go:117] "RemoveContainer" containerID="79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9" May 26 16:31:05.159004 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.158977 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m"] May 26 16:31:05.161527 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.161510 2573 scope.go:117] "RemoveContainer" containerID="a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538" May 26 16:31:05.161784 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:31:05.161766 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538\": container with ID starting with a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538 not found: ID does not exist" containerID="a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538" May 26 16:31:05.161888 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.161797 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538"} err="failed to get container status \"a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538\": rpc error: code = NotFound desc = could not find container \"a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538\": container with ID starting with a575d27ca5b915983ab7b313e364d8d6249c64c947c32589e79453f946352538 not found: ID does not exist" May 26 16:31:05.161888 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.161823 2573 scope.go:117] "RemoveContainer" containerID="1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499" May 26 16:31:05.162123 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:31:05.162101 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499\": container with ID starting with 1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499 not found: ID does not exist" containerID="1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499" May 26 16:31:05.162183 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.162154 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499"} err="failed to get container status \"1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499\": rpc error: code = NotFound desc = could not find container \"1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499\": container with ID starting with 1237c01997fc4084ce4758532e4523790f666915a4f778edb8cc877f4051a499 not found: ID does not exist" May 26 16:31:05.162183 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.162177 2573 scope.go:117] "RemoveContainer" containerID="79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9" May 26 16:31:05.162721 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:31:05.162503 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9\": container with ID starting with 79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9 not found: ID does not exist" containerID="79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9" May 26 16:31:05.162721 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.162537 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9"} err="failed to get container status \"79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9\": rpc error: code = NotFound desc = could not find container \"79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9\": container with ID starting with 79a6b20fdad2b143d3dfe908e9ff6e6f036513ffd5e97206b6a6d7c3508c9da9 not found: ID does not exist" May 26 16:31:05.165499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.165478 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schenlh9m"] May 26 16:31:05.875188 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:05.875154 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" path="/var/lib/kubelet/pods/78f56c9d-8fd4-44a1-991d-6377b51823cc/volumes" May 26 16:31:20.561233 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.561202 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx"] May 26 16:31:20.561693 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.561594 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="main" May 26 16:31:20.561693 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.561606 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="main" May 26 16:31:20.561693 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.561618 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="storage-initializer" May 26 16:31:20.561693 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.561624 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="storage-initializer" May 26 16:31:20.561693 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.561635 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="tokenizer" May 26 16:31:20.561693 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.561640 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="tokenizer" May 26 16:31:20.561926 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.561706 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="tokenizer" May 26 16:31:20.561926 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.561713 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="78f56c9d-8fd4-44a1-991d-6377b51823cc" containerName="main" May 26 16:31:20.566982 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.566961 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.569686 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.569669 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\"" May 26 16:31:20.569770 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.569670 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-fjdh8\"" May 26 16:31:20.576900 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.576879 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx"] May 26 16:31:20.639762 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.639724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.639969 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.639776 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.639969 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.639871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.639969 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.639902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.639969 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.639964 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjx9\" (UniqueName: \"kubernetes.io/projected/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kube-api-access-jzjx9\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.640137 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.639993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.740674 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.740636 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjx9\" (UniqueName: \"kubernetes.io/projected/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kube-api-access-jzjx9\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.740674 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.740678 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.740975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.740724 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.740975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.740748 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.740975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.740790 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.740975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.740853 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.741230 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.741205 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.741306 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.741228 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.741306 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.741267 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.741409 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.741330 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.743244 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.743227 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.748677 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.748657 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjx9\" (UniqueName: \"kubernetes.io/projected/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kube-api-access-jzjx9\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:20.877549 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:20.877463 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:21.016824 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:21.016794 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx"] May 26 16:31:21.018146 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:31:21.018119 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d8a3ea_c30e_4980_b30d_9a73f15f47c2.slice/crio-d348d91ab7b7580a10e08025f423c0398009c788d57c5c595d1ac1f325bde8c8 WatchSource:0}: Error finding container d348d91ab7b7580a10e08025f423c0398009c788d57c5c595d1ac1f325bde8c8: Status 404 returned error can't find the container with id d348d91ab7b7580a10e08025f423c0398009c788d57c5c595d1ac1f325bde8c8 May 26 16:31:21.199945 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:21.199823 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" event={"ID":"44d8a3ea-c30e-4980-b30d-9a73f15f47c2","Type":"ContainerStarted","Data":"567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db"} May 26 16:31:21.199945 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:21.199893 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" event={"ID":"44d8a3ea-c30e-4980-b30d-9a73f15f47c2","Type":"ContainerStarted","Data":"d348d91ab7b7580a10e08025f423c0398009c788d57c5c595d1ac1f325bde8c8"} May 26 16:31:22.205467 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:22.205425 2573 generic.go:358] "Generic (PLEG): container finished" podID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerID="567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db" exitCode=0 May 26 16:31:22.205900 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:22.205499 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" event={"ID":"44d8a3ea-c30e-4980-b30d-9a73f15f47c2","Type":"ContainerDied","Data":"567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db"} May 26 16:31:23.211346 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:23.211305 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" event={"ID":"44d8a3ea-c30e-4980-b30d-9a73f15f47c2","Type":"ContainerStarted","Data":"a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6"} May 26 16:31:23.211346 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:23.211346 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" event={"ID":"44d8a3ea-c30e-4980-b30d-9a73f15f47c2","Type":"ContainerStarted","Data":"2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342"} May 26 16:31:23.211806 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:23.211453 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:23.234357 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:23.234301 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" podStartSLOduration=3.234285851 podStartE2EDuration="3.234285851s" podCreationTimestamp="2026-05-26 16:31:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:31:23.231718028 +0000 UTC m=+1832.091640510" watchObservedRunningTime="2026-05-26 16:31:23.234285851 +0000 UTC m=+1832.094208401" May 26 16:31:30.878302 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:30.878252 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:30.878925 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:30.878427 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:30.881430 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:30.881406 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:31.244757 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:31.244669 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:31:35.131822 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:35.131787 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw"] May 26 16:31:35.132376 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:35.132126 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="main" containerID="cri-o://e43a80569214cf5bdb4216904b89cee81ac513d7a1912a914bb335ee4f36f88f" gracePeriod=30 May 26 16:31:35.132376 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:35.132164 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="tokenizer" containerID="cri-o://19e4808b1dd9a196543c3c1e8dfea5356ce9e55340e5ddd8dad10259cb27e1b2" gracePeriod=30 May 26 16:31:35.260554 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:35.260524 2573 generic.go:358] "Generic (PLEG): container finished" podID="f2e37e30-846e-4286-9088-b95d37293b89" containerID="e43a80569214cf5bdb4216904b89cee81ac513d7a1912a914bb335ee4f36f88f" exitCode=0 May 26 16:31:35.260693 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:35.260600 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" event={"ID":"f2e37e30-846e-4286-9088-b95d37293b89","Type":"ContainerDied","Data":"e43a80569214cf5bdb4216904b89cee81ac513d7a1912a914bb335ee4f36f88f"} May 26 16:31:36.266737 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.266704 2573 generic.go:358] "Generic (PLEG): container finished" podID="f2e37e30-846e-4286-9088-b95d37293b89" containerID="19e4808b1dd9a196543c3c1e8dfea5356ce9e55340e5ddd8dad10259cb27e1b2" exitCode=0 May 26 16:31:36.267129 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.266773 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" event={"ID":"f2e37e30-846e-4286-9088-b95d37293b89","Type":"ContainerDied","Data":"19e4808b1dd9a196543c3c1e8dfea5356ce9e55340e5ddd8dad10259cb27e1b2"} May 26 16:31:36.386529 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.386504 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:31:36.486054 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.485961 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrtxw\" (UniqueName: \"kubernetes.io/projected/f2e37e30-846e-4286-9088-b95d37293b89-kube-api-access-nrtxw\") pod \"f2e37e30-846e-4286-9088-b95d37293b89\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " May 26 16:31:36.486054 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.486019 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-kserve-provision-location\") pod \"f2e37e30-846e-4286-9088-b95d37293b89\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " May 26 16:31:36.486054 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.486042 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-tmp\") pod \"f2e37e30-846e-4286-9088-b95d37293b89\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " May 26 16:31:36.486392 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.486084 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-cache\") pod \"f2e37e30-846e-4286-9088-b95d37293b89\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " May 26 16:31:36.486392 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.486125 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e37e30-846e-4286-9088-b95d37293b89-tls-certs\") pod \"f2e37e30-846e-4286-9088-b95d37293b89\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " May 26 16:31:36.486392 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.486373 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f2e37e30-846e-4286-9088-b95d37293b89" (UID: "f2e37e30-846e-4286-9088-b95d37293b89"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:31:36.486552 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.486389 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f2e37e30-846e-4286-9088-b95d37293b89" (UID: "f2e37e30-846e-4286-9088-b95d37293b89"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:31:36.486680 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.486658 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2e37e30-846e-4286-9088-b95d37293b89" (UID: "f2e37e30-846e-4286-9088-b95d37293b89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:31:36.488194 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.488173 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e37e30-846e-4286-9088-b95d37293b89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2e37e30-846e-4286-9088-b95d37293b89" (UID: "f2e37e30-846e-4286-9088-b95d37293b89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:31:36.488300 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.488284 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e37e30-846e-4286-9088-b95d37293b89-kube-api-access-nrtxw" (OuterVolumeSpecName: "kube-api-access-nrtxw") pod "f2e37e30-846e-4286-9088-b95d37293b89" (UID: "f2e37e30-846e-4286-9088-b95d37293b89"). InnerVolumeSpecName "kube-api-access-nrtxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:31:36.586624 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.586587 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-uds\") pod \"f2e37e30-846e-4286-9088-b95d37293b89\" (UID: \"f2e37e30-846e-4286-9088-b95d37293b89\") " May 26 16:31:36.586822 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.586811 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtxw\" (UniqueName: \"kubernetes.io/projected/f2e37e30-846e-4286-9088-b95d37293b89-kube-api-access-nrtxw\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:36.586901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.586825 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:36.586901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.586860 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f2e37e30-846e-4286-9088-b95d37293b89" (UID: "f2e37e30-846e-4286-9088-b95d37293b89"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:31:36.586901 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.586869 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:36.587007 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.586912 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:36.587007 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.586923 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e37e30-846e-4286-9088-b95d37293b89-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:36.688005 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:36.687965 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2e37e30-846e-4286-9088-b95d37293b89-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:31:37.272985 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:37.272945 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" event={"ID":"f2e37e30-846e-4286-9088-b95d37293b89","Type":"ContainerDied","Data":"478896eaab510037cedb6df7b86e9066048261c21b1f5d0bd08306738940bd6a"} May 26 16:31:37.272985 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:37.272988 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw" May 26 16:31:37.273448 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:37.272996 2573 scope.go:117] "RemoveContainer" containerID="19e4808b1dd9a196543c3c1e8dfea5356ce9e55340e5ddd8dad10259cb27e1b2" May 26 16:31:37.283146 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:37.283129 2573 scope.go:117] "RemoveContainer" containerID="e43a80569214cf5bdb4216904b89cee81ac513d7a1912a914bb335ee4f36f88f" May 26 16:31:37.292290 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:37.292268 2573 scope.go:117] "RemoveContainer" containerID="de96c7f92aeb39034aa60b28180a1836db101fd5e7f488db5ada3098def9cca8" May 26 16:31:37.298072 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:37.297993 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw"] May 26 16:31:37.300798 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:37.300776 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945drmcfw"] May 26 16:31:37.874939 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:37.874908 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e37e30-846e-4286-9088-b95d37293b89" path="/var/lib/kubelet/pods/f2e37e30-846e-4286-9088-b95d37293b89/volumes" May 26 16:31:45.817468 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.817433 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6"] May 26 16:31:45.817885 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.817871 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="storage-initializer" May 26 16:31:45.817942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.817886 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="storage-initializer" May 26 16:31:45.817942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.817897 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="tokenizer" May 26 16:31:45.817942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.817903 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="tokenizer" May 26 16:31:45.817942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.817921 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="main" May 26 16:31:45.817942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.817926 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="main" May 26 16:31:45.818101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.817990 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="main" May 26 16:31:45.818101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.818000 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2e37e30-846e-4286-9088-b95d37293b89" containerName="tokenizer" May 26 16:31:45.823132 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.823114 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.825617 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.825593 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\"" May 26 16:31:45.832819 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.832794 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6"] May 26 16:31:45.872924 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.872892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tls-certs\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.873122 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.872942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-home\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.873122 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.873001 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-model-cache\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.873122 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.873058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-dshm\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.873122 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.873080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.873313 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.873219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.873313 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.873289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6dx\" (UniqueName: \"kubernetes.io/projected/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kube-api-access-hp6dx\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974327 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974272 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974543 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974361 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6dx\" (UniqueName: \"kubernetes.io/projected/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kube-api-access-hp6dx\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974543 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974414 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tls-certs\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974543 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974450 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-home\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974543 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974494 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-model-cache\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974543 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974531 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-dshm\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974811 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974561 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974811 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974767 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974973 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974817 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-home\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.974973 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974877 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-model-cache\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.975065 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.974980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.976909 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.976888 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-dshm\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.977267 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.977244 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tls-certs\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:45.982396 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:45.982367 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6dx\" (UniqueName: \"kubernetes.io/projected/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kube-api-access-hp6dx\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-sg8q6\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:46.134790 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:46.134704 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:31:46.273139 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:46.273114 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6"] May 26 16:31:46.274856 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:31:46.274815 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cd47dd_aaa4_4527_ad53_5a3ef1441f91.slice/crio-251f9f5bbf1155070ae70597de267015222ce4f82532ff5722f8b6bbba7c749a WatchSource:0}: Error finding container 251f9f5bbf1155070ae70597de267015222ce4f82532ff5722f8b6bbba7c749a: Status 404 returned error can't find the container with id 251f9f5bbf1155070ae70597de267015222ce4f82532ff5722f8b6bbba7c749a May 26 16:31:46.310547 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:46.310514 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" event={"ID":"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91","Type":"ContainerStarted","Data":"251f9f5bbf1155070ae70597de267015222ce4f82532ff5722f8b6bbba7c749a"} May 26 16:31:47.316625 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:47.316582 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" event={"ID":"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91","Type":"ContainerStarted","Data":"7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740"} May 26 16:31:53.253084 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:31:53.253054 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:33:27.745925 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:27.745891 2573 generic.go:358] "Generic (PLEG): container finished" podID="a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" containerID="7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740" exitCode=0 May 26 16:33:27.746351 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:27.745965 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" event={"ID":"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91","Type":"ContainerDied","Data":"7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740"} May 26 16:33:27.747141 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:27.747054 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:33:29.755999 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:29.755965 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" event={"ID":"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91","Type":"ContainerStarted","Data":"acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a"} May 26 16:33:29.777672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:29.777623 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" podStartSLOduration=103.440550648 podStartE2EDuration="1m44.777608899s" podCreationTimestamp="2026-05-26 16:31:45 +0000 UTC" firstStartedPulling="2026-05-26 16:33:27.747171773 +0000 UTC m=+1956.607094219" lastFinishedPulling="2026-05-26 16:33:29.084230006 +0000 UTC m=+1957.944152470" observedRunningTime="2026-05-26 16:33:29.776880046 +0000 UTC m=+1958.636802514" watchObservedRunningTime="2026-05-26 16:33:29.777608899 +0000 UTC m=+1958.637531368" May 26 16:33:36.134942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:36.134907 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:33:36.134942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:36.134947 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:33:36.147763 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:36.147735 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:33:36.797125 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:36.797100 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:33:38.309480 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:38.309448 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6"] May 26 16:33:38.794183 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:38.794136 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" podUID="a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" containerName="main" containerID="cri-o://acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a" gracePeriod=30 May 26 16:33:39.094672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.094644 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:33:39.231275 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231239 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tls-certs\") pod \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " May 26 16:33:39.231489 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231302 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-model-cache\") pod \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " May 26 16:33:39.231489 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231325 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-dshm\") pod \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " May 26 16:33:39.231489 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231365 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp6dx\" (UniqueName: \"kubernetes.io/projected/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kube-api-access-hp6dx\") pod \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " May 26 16:33:39.231489 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231416 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tmp-dir\") pod \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " May 26 16:33:39.231489 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231457 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kserve-provision-location\") pod \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " May 26 16:33:39.231753 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231498 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-home\") pod \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\" (UID: \"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91\") " May 26 16:33:39.231753 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231604 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" (UID: "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:33:39.231753 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231685 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" (UID: "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:33:39.231923 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231781 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-home" (OuterVolumeSpecName: "home") pod "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" (UID: "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:33:39.231923 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231871 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tmp-dir\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:33:39.231923 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.231894 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-model-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:33:39.233575 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.233541 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" (UID: "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:33:39.233701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.233600 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-dshm" (OuterVolumeSpecName: "dshm") pod "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" (UID: "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:33:39.233815 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.233794 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kube-api-access-hp6dx" (OuterVolumeSpecName: "kube-api-access-hp6dx") pod "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" (UID: "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91"). InnerVolumeSpecName "kube-api-access-hp6dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:33:39.332821 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.332648 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp6dx\" (UniqueName: \"kubernetes.io/projected/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kube-api-access-hp6dx\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:33:39.332821 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.332695 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-home\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:33:39.332821 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.332711 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:33:39.332821 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.332726 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-dshm\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:33:39.805546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.805507 2573 generic.go:358] "Generic (PLEG): container finished" podID="a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" containerID="acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a" exitCode=0 May 26 16:33:39.805758 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.805594 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" May 26 16:33:39.805758 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.805594 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" event={"ID":"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91","Type":"ContainerDied","Data":"acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a"} May 26 16:33:39.805758 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.805639 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6" event={"ID":"a2cd47dd-aaa4-4527-ad53-5a3ef1441f91","Type":"ContainerDied","Data":"251f9f5bbf1155070ae70597de267015222ce4f82532ff5722f8b6bbba7c749a"} May 26 16:33:39.805758 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.805661 2573 scope.go:117] "RemoveContainer" containerID="acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a" May 26 16:33:39.814862 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.814843 2573 scope.go:117] "RemoveContainer" containerID="7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740" May 26 16:33:39.875295 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.875266 2573 scope.go:117] "RemoveContainer" containerID="acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a" May 26 16:33:39.875585 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:33:39.875562 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a\": container with ID starting with acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a not found: ID does not exist" containerID="acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a" May 26 16:33:39.875683 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.875598 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a"} err="failed to get container status \"acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a\": rpc error: code = NotFound desc = could not find container \"acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a\": container with ID starting with acd3daed0ad20d1fcdfb51d4f9b02ab65682cad67039c606698a05ea5584964a not found: ID does not exist" May 26 16:33:39.875683 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.875622 2573 scope.go:117] "RemoveContainer" containerID="7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740" May 26 16:33:39.875903 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:33:39.875885 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740\": container with ID starting with 7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740 not found: ID does not exist" containerID="7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740" May 26 16:33:39.875967 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.875909 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740"} err="failed to get container status \"7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740\": rpc error: code = NotFound desc = could not find container \"7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740\": container with ID starting with 7743512147c63050249f1296f33ed7402fec921ca2bccca96ff2c0ae882b2740 not found: ID does not exist" May 26 16:33:39.961227 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:39.961188 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" (UID: "a2cd47dd-aaa4-4527-ad53-5a3ef1441f91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:33:40.039862 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:40.039796 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:33:40.135131 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:40.135103 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6"] May 26 16:33:40.137440 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:40.137419 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-sg8q6"] May 26 16:33:41.874929 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:33:41.874820 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" path="/var/lib/kubelet/pods/a2cd47dd-aaa4-4527-ad53-5a3ef1441f91/volumes" May 26 16:34:42.883109 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.883076 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx"] May 26 16:34:42.883651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.883393 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="main" containerID="cri-o://2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342" gracePeriod=30 May 26 16:34:42.883651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.883420 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="tokenizer" containerID="cri-o://a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6" gracePeriod=30 May 26 16:34:42.916499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.916465 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf"] May 26 16:34:42.917018 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.917000 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" containerName="storage-initializer" May 26 16:34:42.917149 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.917021 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" containerName="storage-initializer" May 26 16:34:42.917149 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.917040 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" containerName="main" May 26 16:34:42.917149 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.917048 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" containerName="main" May 26 16:34:42.917362 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.917163 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2cd47dd-aaa4-4527-ad53-5a3ef1441f91" containerName="main" May 26 16:34:42.920505 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.920481 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:42.923195 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.923160 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\"" May 26 16:34:42.923331 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.923166 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-z4ztl\"" May 26 16:34:42.935966 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:42.935446 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf"] May 26 16:34:43.000666 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.000641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17698f4-89e1-4876-babd-e3f49cd03eba-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.000744 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.000682 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdsk\" (UniqueName: \"kubernetes.io/projected/e17698f4-89e1-4876-babd-e3f49cd03eba-kube-api-access-5mdsk\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.000824 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.000802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.001079 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.001059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.001130 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.001097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.001181 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.001159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.001236 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.001183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.001236 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.001211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.001348 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.001238 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.073457 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.073422 2573 generic.go:358] "Generic (PLEG): container finished" podID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerID="2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342" exitCode=0 May 26 16:34:43.073605 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.073463 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" event={"ID":"44d8a3ea-c30e-4980-b30d-9a73f15f47c2","Type":"ContainerDied","Data":"2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342"} May 26 16:34:43.102117 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102073 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102320 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102128 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102320 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102159 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102437 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102323 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17698f4-89e1-4876-babd-e3f49cd03eba-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102437 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102367 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdsk\" (UniqueName: \"kubernetes.io/projected/e17698f4-89e1-4876-babd-e3f49cd03eba-kube-api-access-5mdsk\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102437 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102403 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102600 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102470 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102600 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102509 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102600 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102556 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102600 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102579 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102817 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102609 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102918 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102884 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.102985 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.102922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.103070 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.103041 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17698f4-89e1-4876-babd-e3f49cd03eba-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.104981 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.104954 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.105296 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.105272 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.111464 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.111436 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdsk\" (UniqueName: \"kubernetes.io/projected/e17698f4-89e1-4876-babd-e3f49cd03eba-kube-api-access-5mdsk\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.111966 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.111945 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17698f4-89e1-4876-babd-e3f49cd03eba-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-rj2lf\" (UID: \"e17698f4-89e1-4876-babd-e3f49cd03eba\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.239814 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.239720 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:43.252098 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:34:43.252068 2573 logging.go:55] [core] [Channel #1156 SubChannel #1157]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.55:9003", ServerName: "10.133.0.55:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.55:9003: connect: connection refused" May 26 16:34:43.392811 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:43.392743 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf"] May 26 16:34:43.395261 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:34:43.395228 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17698f4_89e1_4876_babd_e3f49cd03eba.slice/crio-7c9c600576424d19544041144f5e99e50fd946936315e0602eda0addaec52fcb WatchSource:0}: Error finding container 7c9c600576424d19544041144f5e99e50fd946936315e0602eda0addaec52fcb: Status 404 returned error can't find the container with id 7c9c600576424d19544041144f5e99e50fd946936315e0602eda0addaec52fcb May 26 16:34:44.079133 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.079097 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" event={"ID":"e17698f4-89e1-4876-babd-e3f49cd03eba","Type":"ContainerStarted","Data":"7c9c600576424d19544041144f5e99e50fd946936315e0602eda0addaec52fcb"} May 26 16:34:44.244785 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.244757 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:34:44.252751 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.252455 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:9003\" within 1s: context deadline exceeded" May 26 16:34:44.314518 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314431 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-cache\") pod \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " May 26 16:34:44.314518 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314482 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzjx9\" (UniqueName: \"kubernetes.io/projected/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kube-api-access-jzjx9\") pod \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " May 26 16:34:44.314706 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314526 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-uds\") pod \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " May 26 16:34:44.314706 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314574 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tls-certs\") pod \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " May 26 16:34:44.314706 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314631 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-tmp\") pod \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " May 26 16:34:44.314706 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314695 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kserve-provision-location\") pod \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\" (UID: \"44d8a3ea-c30e-4980-b30d-9a73f15f47c2\") " May 26 16:34:44.314706 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314699 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "44d8a3ea-c30e-4980-b30d-9a73f15f47c2" (UID: "44d8a3ea-c30e-4980-b30d-9a73f15f47c2"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:34:44.315019 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314794 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "44d8a3ea-c30e-4980-b30d-9a73f15f47c2" (UID: "44d8a3ea-c30e-4980-b30d-9a73f15f47c2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:34:44.315019 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314985 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:34:44.315019 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.314998 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:34:44.315151 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.315014 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "44d8a3ea-c30e-4980-b30d-9a73f15f47c2" (UID: "44d8a3ea-c30e-4980-b30d-9a73f15f47c2"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:34:44.315421 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.315394 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44d8a3ea-c30e-4980-b30d-9a73f15f47c2" (UID: "44d8a3ea-c30e-4980-b30d-9a73f15f47c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:34:44.317320 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.317286 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44d8a3ea-c30e-4980-b30d-9a73f15f47c2" (UID: "44d8a3ea-c30e-4980-b30d-9a73f15f47c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:34:44.317320 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.317313 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kube-api-access-jzjx9" (OuterVolumeSpecName: "kube-api-access-jzjx9") pod "44d8a3ea-c30e-4980-b30d-9a73f15f47c2" (UID: "44d8a3ea-c30e-4980-b30d-9a73f15f47c2"). InnerVolumeSpecName "kube-api-access-jzjx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:34:44.415940 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.415907 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjx9\" (UniqueName: \"kubernetes.io/projected/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kube-api-access-jzjx9\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:34:44.415940 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.415936 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:34:44.415940 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.415946 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:34:44.416168 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:44.415955 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d8a3ea-c30e-4980-b30d-9a73f15f47c2-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:34:45.086585 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.086546 2573 generic.go:358] "Generic (PLEG): container finished" podID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerID="a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6" exitCode=0 May 26 16:34:45.086585 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.086574 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" May 26 16:34:45.086585 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.086577 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" event={"ID":"44d8a3ea-c30e-4980-b30d-9a73f15f47c2","Type":"ContainerDied","Data":"a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6"} May 26 16:34:45.087214 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.086623 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx" event={"ID":"44d8a3ea-c30e-4980-b30d-9a73f15f47c2","Type":"ContainerDied","Data":"d348d91ab7b7580a10e08025f423c0398009c788d57c5c595d1ac1f325bde8c8"} May 26 16:34:45.087214 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.086645 2573 scope.go:117] "RemoveContainer" containerID="a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6" May 26 16:34:45.098761 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.098739 2573 scope.go:117] "RemoveContainer" containerID="2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342" May 26 16:34:45.109266 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.109054 2573 scope.go:117] "RemoveContainer" containerID="567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db" May 26 16:34:45.116440 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.115983 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx"] May 26 16:34:45.120304 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.120259 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c45f668jx"] May 26 16:34:45.124101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.123023 2573 scope.go:117] "RemoveContainer" containerID="a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6" May 26 16:34:45.124101 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:34:45.123326 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6\": container with ID starting with a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6 not found: ID does not exist" containerID="a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6" May 26 16:34:45.124101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.123358 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6"} err="failed to get container status \"a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6\": rpc error: code = NotFound desc = could not find container \"a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6\": container with ID starting with a0200ea2e0d5cb4bd1739b5997e9792216b0afa232d9ff9a965cd24182c05bb6 not found: ID does not exist" May 26 16:34:45.124101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.123382 2573 scope.go:117] "RemoveContainer" containerID="2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342" May 26 16:34:45.124101 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:34:45.123592 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342\": container with ID starting with 2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342 not found: ID does not exist" containerID="2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342" May 26 16:34:45.124101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.123614 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342"} err="failed to get container status \"2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342\": rpc error: code = NotFound desc = could not find container \"2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342\": container with ID starting with 2c58dfe4c57229e7324b3091f9337252fc4c086886617a179664e0164d21a342 not found: ID does not exist" May 26 16:34:45.124101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.123629 2573 scope.go:117] "RemoveContainer" containerID="567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db" May 26 16:34:45.124101 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:34:45.123854 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db\": container with ID starting with 567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db not found: ID does not exist" containerID="567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db" May 26 16:34:45.124101 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.123879 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db"} err="failed to get container status \"567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db\": rpc error: code = NotFound desc = could not find container \"567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db\": container with ID starting with 567abe343e1124d9ada76b9631b6936b1a94525944018cea9e8a2934225de4db not found: ID does not exist" May 26 16:34:45.881322 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:45.881286 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" path="/var/lib/kubelet/pods/44d8a3ea-c30e-4980-b30d-9a73f15f47c2/volumes" May 26 16:34:46.249546 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:46.249504 2573 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} May 26 16:34:46.249812 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:46.249596 2573 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} May 26 16:34:46.249812 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:46.249640 2573 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} May 26 16:34:47.098624 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:47.098589 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" event={"ID":"e17698f4-89e1-4876-babd-e3f49cd03eba","Type":"ContainerStarted","Data":"01c969e897647f2610f8f6fdc093bf7b55320680e034a39af680f6c7ff7015e8"} May 26 16:34:47.122882 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:47.122806 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" podStartSLOduration=2.270608536 podStartE2EDuration="5.122759401s" podCreationTimestamp="2026-05-26 16:34:42 +0000 UTC" firstStartedPulling="2026-05-26 16:34:43.39713838 +0000 UTC m=+2032.257060828" lastFinishedPulling="2026-05-26 16:34:46.249289232 +0000 UTC m=+2035.109211693" observedRunningTime="2026-05-26 16:34:47.12116564 +0000 UTC m=+2035.981088109" watchObservedRunningTime="2026-05-26 16:34:47.122759401 +0000 UTC m=+2035.982681870" May 26 16:34:47.240665 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:47.240580 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:47.241980 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:47.241954 2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" podUID="e17698f4-89e1-4876-babd-e3f49cd03eba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" May 26 16:34:48.240382 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:48.240341 2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" podUID="e17698f4-89e1-4876-babd-e3f49cd03eba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" May 26 16:34:49.240760 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:49.240717 2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" podUID="e17698f4-89e1-4876-babd-e3f49cd03eba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" May 26 16:34:50.244783 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:50.244754 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:50.245223 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:50.245144 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:34:50.246051 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:34:50.246034 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-rj2lf" May 26 16:35:01.073953 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.073905 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh"] May 26 16:35:01.074567 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.074548 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="main" May 26 16:35:01.074567 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.074568 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="main" May 26 16:35:01.074713 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.074591 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="storage-initializer" May 26 16:35:01.074713 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.074601 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="storage-initializer" May 26 16:35:01.074713 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.074623 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="tokenizer" May 26 16:35:01.074713 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.074632 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="tokenizer" May 26 16:35:01.074982 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.074718 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="main" May 26 16:35:01.074982 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.074737 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="44d8a3ea-c30e-4980-b30d-9a73f15f47c2" containerName="tokenizer" May 26 16:35:01.078643 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.078620 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.082288 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.082261 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\"" May 26 16:35:01.082412 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.082309 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-gwfk2\"" May 26 16:35:01.082577 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.082558 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\"" May 26 16:35:01.085727 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.085696 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh"] May 26 16:35:01.187713 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.187658 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c20829-426c-436e-8cad-dcdbd17e3bfc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.187713 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.187722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.187994 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.187876 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxj9\" (UniqueName: \"kubernetes.io/projected/59c20829-426c-436e-8cad-dcdbd17e3bfc-kube-api-access-qmxj9\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.187994 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.187918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.187994 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.187966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.188097 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.188000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.288844 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.288790 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.289027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.288867 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c20829-426c-436e-8cad-dcdbd17e3bfc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.289027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.288903 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.289027 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.288992 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxj9\" (UniqueName: \"kubernetes.io/projected/59c20829-426c-436e-8cad-dcdbd17e3bfc-kube-api-access-qmxj9\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.289175 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.289137 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.289251 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.289231 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.289306 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.289254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.289355 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.289327 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.289499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.289474 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.289545 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.289507 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.291376 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.291351 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c20829-426c-436e-8cad-dcdbd17e3bfc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.297296 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.297274 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxj9\" (UniqueName: \"kubernetes.io/projected/59c20829-426c-436e-8cad-dcdbd17e3bfc-kube-api-access-qmxj9\") pod \"router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.390647 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.390541 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:01.530002 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:01.529973 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh"] May 26 16:35:01.531331 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:35:01.531302 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c20829_426c_436e_8cad_dcdbd17e3bfc.slice/crio-903910f110352e665531a6b3a243725b41195933b25c9e3e9a1cf2b9a76072de WatchSource:0}: Error finding container 903910f110352e665531a6b3a243725b41195933b25c9e3e9a1cf2b9a76072de: Status 404 returned error can't find the container with id 903910f110352e665531a6b3a243725b41195933b25c9e3e9a1cf2b9a76072de May 26 16:35:02.165471 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:02.165433 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" event={"ID":"59c20829-426c-436e-8cad-dcdbd17e3bfc","Type":"ContainerStarted","Data":"da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e"} May 26 16:35:02.165899 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:02.165478 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" event={"ID":"59c20829-426c-436e-8cad-dcdbd17e3bfc","Type":"ContainerStarted","Data":"903910f110352e665531a6b3a243725b41195933b25c9e3e9a1cf2b9a76072de"} May 26 16:35:03.171289 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:03.171253 2573 generic.go:358] "Generic (PLEG): container finished" podID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerID="da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e" exitCode=0 May 26 16:35:03.171685 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:03.171343 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" event={"ID":"59c20829-426c-436e-8cad-dcdbd17e3bfc","Type":"ContainerDied","Data":"da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e"} May 26 16:35:04.177076 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:04.177038 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" event={"ID":"59c20829-426c-436e-8cad-dcdbd17e3bfc","Type":"ContainerStarted","Data":"ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b"} May 26 16:35:04.177076 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:04.177078 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" event={"ID":"59c20829-426c-436e-8cad-dcdbd17e3bfc","Type":"ContainerStarted","Data":"6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2"} May 26 16:35:04.177639 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:04.177124 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:04.204167 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:04.204091 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" podStartSLOduration=3.204073238 podStartE2EDuration="3.204073238s" podCreationTimestamp="2026-05-26 16:35:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:35:04.200944963 +0000 UTC m=+2053.060867443" watchObservedRunningTime="2026-05-26 16:35:04.204073238 +0000 UTC m=+2053.063995708" May 26 16:35:11.391651 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:11.391549 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:11.392155 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:11.391715 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:11.394572 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:11.394546 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:12.212800 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:12.212776 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:34.219948 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:34.219913 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:35:51.964539 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:51.964511 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:35:51.977052 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:35:51.977031 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:37:32.967089 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:32.967047 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh"] May 26 16:37:32.967664 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:32.967463 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="main" containerID="cri-o://6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2" gracePeriod=30 May 26 16:37:32.967664 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:32.967502 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="tokenizer" containerID="cri-o://ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b" gracePeriod=30 May 26 16:37:33.791675 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:33.791638 2573 generic.go:358] "Generic (PLEG): container finished" podID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerID="6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2" exitCode=0 May 26 16:37:33.791894 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:33.791712 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" event={"ID":"59c20829-426c-436e-8cad-dcdbd17e3bfc","Type":"ContainerDied","Data":"6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2"} May 26 16:37:34.227070 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.227040 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:37:34.288075 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288045 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c20829-426c-436e-8cad-dcdbd17e3bfc-tls-certs\") pod \"59c20829-426c-436e-8cad-dcdbd17e3bfc\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " May 26 16:37:34.288279 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288086 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-kserve-provision-location\") pod \"59c20829-426c-436e-8cad-dcdbd17e3bfc\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " May 26 16:37:34.288279 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288155 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-tmp\") pod \"59c20829-426c-436e-8cad-dcdbd17e3bfc\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " May 26 16:37:34.288279 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288205 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmxj9\" (UniqueName: \"kubernetes.io/projected/59c20829-426c-436e-8cad-dcdbd17e3bfc-kube-api-access-qmxj9\") pod \"59c20829-426c-436e-8cad-dcdbd17e3bfc\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " May 26 16:37:34.288279 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288228 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-uds\") pod \"59c20829-426c-436e-8cad-dcdbd17e3bfc\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " May 26 16:37:34.288510 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288353 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-cache\") pod \"59c20829-426c-436e-8cad-dcdbd17e3bfc\" (UID: \"59c20829-426c-436e-8cad-dcdbd17e3bfc\") " May 26 16:37:34.288510 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288472 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "59c20829-426c-436e-8cad-dcdbd17e3bfc" (UID: "59c20829-426c-436e-8cad-dcdbd17e3bfc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:37:34.288608 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288527 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "59c20829-426c-436e-8cad-dcdbd17e3bfc" (UID: "59c20829-426c-436e-8cad-dcdbd17e3bfc"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:37:34.288654 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288604 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "59c20829-426c-436e-8cad-dcdbd17e3bfc" (UID: "59c20829-426c-436e-8cad-dcdbd17e3bfc"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:37:34.288783 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288768 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:37:34.288856 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288787 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:37:34.288856 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288800 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:37:34.289005 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.288980 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59c20829-426c-436e-8cad-dcdbd17e3bfc" (UID: "59c20829-426c-436e-8cad-dcdbd17e3bfc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:37:34.290351 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.290333 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c20829-426c-436e-8cad-dcdbd17e3bfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59c20829-426c-436e-8cad-dcdbd17e3bfc" (UID: "59c20829-426c-436e-8cad-dcdbd17e3bfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:37:34.290442 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.290420 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c20829-426c-436e-8cad-dcdbd17e3bfc-kube-api-access-qmxj9" (OuterVolumeSpecName: "kube-api-access-qmxj9") pod "59c20829-426c-436e-8cad-dcdbd17e3bfc" (UID: "59c20829-426c-436e-8cad-dcdbd17e3bfc"). InnerVolumeSpecName "kube-api-access-qmxj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:37:34.389896 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.389799 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxj9\" (UniqueName: \"kubernetes.io/projected/59c20829-426c-436e-8cad-dcdbd17e3bfc-kube-api-access-qmxj9\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:37:34.389896 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.389823 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c20829-426c-436e-8cad-dcdbd17e3bfc-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:37:34.389896 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.389849 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c20829-426c-436e-8cad-dcdbd17e3bfc-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:37:34.797500 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.797465 2573 generic.go:358] "Generic (PLEG): container finished" podID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerID="ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b" exitCode=0 May 26 16:37:34.797656 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.797550 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" event={"ID":"59c20829-426c-436e-8cad-dcdbd17e3bfc","Type":"ContainerDied","Data":"ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b"} May 26 16:37:34.797656 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.797571 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" May 26 16:37:34.797656 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.797590 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" event={"ID":"59c20829-426c-436e-8cad-dcdbd17e3bfc","Type":"ContainerDied","Data":"903910f110352e665531a6b3a243725b41195933b25c9e3e9a1cf2b9a76072de"} May 26 16:37:34.797656 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.797609 2573 scope.go:117] "RemoveContainer" containerID="ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b" May 26 16:37:34.807512 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.807495 2573 scope.go:117] "RemoveContainer" containerID="6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2" May 26 16:37:34.815472 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.815453 2573 scope.go:117] "RemoveContainer" containerID="da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e" May 26 16:37:34.822891 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.822868 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh"] May 26 16:37:34.824001 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.823982 2573 scope.go:117] "RemoveContainer" containerID="ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b" May 26 16:37:34.824375 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:37:34.824347 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b\": container with ID starting with ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b not found: ID does not exist" containerID="ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b" May 26 16:37:34.824520 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.824387 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b"} err="failed to get container status \"ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b\": rpc error: code = NotFound desc = could not find container \"ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b\": container with ID starting with ff5d287fd33110174fe29573235be8ed9ce41d4e3b155cb500aa29a2a991ef8b not found: ID does not exist" May 26 16:37:34.824520 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.824419 2573 scope.go:117] "RemoveContainer" containerID="6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2" May 26 16:37:34.824909 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:37:34.824886 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2\": container with ID starting with 6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2 not found: ID does not exist" containerID="6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2" May 26 16:37:34.825001 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.824925 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2"} err="failed to get container status \"6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2\": rpc error: code = NotFound desc = could not find container \"6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2\": container with ID starting with 6b38838d25dc708b6ba76bdbd85e975b16e5d7d1750a8e9ea23d9df2f15b6fd2 not found: ID does not exist" May 26 16:37:34.825001 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.824949 2573 scope.go:117] "RemoveContainer" containerID="da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e" May 26 16:37:34.825228 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:37:34.825212 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e\": container with ID starting with da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e not found: ID does not exist" containerID="da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e" May 26 16:37:34.825288 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.825233 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e"} err="failed to get container status \"da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e\": rpc error: code = NotFound desc = could not find container \"da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e\": container with ID starting with da288fe4ffd0325901c7916d3f80dec60cff048478f4dfeaafa18a30a788967e not found: ID does not exist" May 26 16:37:34.827177 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:34.827158 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh"] May 26 16:37:35.219222 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:35.219110 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-65bd8597-sm9wh" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.58:9003\" within 1s: context deadline exceeded" May 26 16:37:35.219222 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:37:35.219224 2573 logging.go:55] [core] [Channel #1239 SubChannel #1240]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.58:9003", ServerName: "10.133.0.58:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.58:9003: operation was canceled" May 26 16:37:35.874322 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:37:35.874289 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" path="/var/lib/kubelet/pods/59c20829-426c-436e-8cad-dcdbd17e3bfc/volumes" May 26 16:39:34.788147 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.788108 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] May 26 16:39:34.788609 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.788511 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="main" May 26 16:39:34.788609 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.788523 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="main" May 26 16:39:34.788609 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.788535 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="storage-initializer" May 26 16:39:34.788609 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.788541 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="storage-initializer" May 26 16:39:34.788609 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.788547 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="tokenizer" May 26 16:39:34.788609 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.788553 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="tokenizer" May 26 16:39:34.788873 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.788615 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="tokenizer" May 26 16:39:34.788873 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.788625 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="59c20829-426c-436e-8cad-dcdbd17e3bfc" containerName="main" May 26 16:39:34.791987 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.791967 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.795124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.795097 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\"" May 26 16:39:34.795124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.795103 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\"" May 26 16:39:34.795124 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.795124 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-2m5pn\"" May 26 16:39:34.802325 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.802302 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] May 26 16:39:34.855433 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.855403 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2"] May 26 16:39:34.859240 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.859218 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:34.861965 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.861942 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa-dockercfg-w9xfd\"" May 26 16:39:34.872128 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.872106 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2"] May 26 16:39:34.897223 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.897199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.897368 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.897255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.897368 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.897342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.897457 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.897377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.897457 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.897431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.897538 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.897460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.897538 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.897502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nplj\" (UniqueName: \"kubernetes.io/projected/09b5c3fc-0887-4402-aff2-92600665b90b-kube-api-access-7nplj\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998031 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.997988 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:34.998247 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998049 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998247 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:34.998247 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:34.998247 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8tvk\" (UniqueName: \"kubernetes.io/projected/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kube-api-access-k8tvk\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:34.998455 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998270 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998455 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:34.998455 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998432 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998467 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:34.998597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998501 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998532 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998550 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998597 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998579 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998876 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998644 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7nplj\" (UniqueName: \"kubernetes.io/projected/09b5c3fc-0887-4402-aff2-92600665b90b-kube-api-access-7nplj\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998876 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998752 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998962 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998874 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:34.998962 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:34.998909 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:35.000453 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.000428 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:35.000773 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.000757 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:35.007212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.007183 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nplj\" (UniqueName: \"kubernetes.io/projected/09b5c3fc-0887-4402-aff2-92600665b90b-kube-api-access-7nplj\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:35.100268 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100170 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.100268 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100229 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.100505 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100328 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.100505 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100372 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.100625 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100500 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.100625 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100546 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k8tvk\" (UniqueName: \"kubernetes.io/projected/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kube-api-access-k8tvk\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.100625 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100551 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.100777 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100639 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.100777 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100695 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.100897 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.100820 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.102795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.102769 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.102957 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.102920 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:39:35.108414 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.108392 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8tvk\" (UniqueName: \"kubernetes.io/projected/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kube-api-access-k8tvk\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.169961 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.169927 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:35.244488 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.244458 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] May 26 16:39:35.245750 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:39:35.245724 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b5c3fc_0887_4402_aff2_92600665b90b.slice/crio-a698bc16758807e2fe4cab0bcdfebdc1767c58e699c750aa8043f2f7f923363c WatchSource:0}: Error finding container a698bc16758807e2fe4cab0bcdfebdc1767c58e699c750aa8043f2f7f923363c: Status 404 returned error can't find the container with id a698bc16758807e2fe4cab0bcdfebdc1767c58e699c750aa8043f2f7f923363c May 26 16:39:35.248288 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.248268 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 26 16:39:35.277499 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.277464 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"09b5c3fc-0887-4402-aff2-92600665b90b","Type":"ContainerStarted","Data":"a698bc16758807e2fe4cab0bcdfebdc1767c58e699c750aa8043f2f7f923363c"} May 26 16:39:35.311645 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:35.311618 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2"] May 26 16:39:35.312643 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:39:35.312618 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25bbdb8_8bc9_41b7_9b37_8d194ef50087.slice/crio-0d6bb4632ec2c4e04bd746e9224a3ee63b2f3abf22baf92856d6d794c8efcf9d WatchSource:0}: Error finding container 0d6bb4632ec2c4e04bd746e9224a3ee63b2f3abf22baf92856d6d794c8efcf9d: Status 404 returned error can't find the container with id 0d6bb4632ec2c4e04bd746e9224a3ee63b2f3abf22baf92856d6d794c8efcf9d May 26 16:39:36.282725 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:36.282633 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"09b5c3fc-0887-4402-aff2-92600665b90b","Type":"ContainerStarted","Data":"4e58418afcf1e26be6a33aa724f708704c8a9935ce25b9894a4b5a8cefff7f02"} May 26 16:39:36.284082 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:36.284056 2573 generic.go:358] "Generic (PLEG): container finished" podID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerID="a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7" exitCode=0 May 26 16:39:36.284184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:36.284103 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" event={"ID":"b25bbdb8-8bc9-41b7-9b37-8d194ef50087","Type":"ContainerDied","Data":"a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7"} May 26 16:39:36.284184 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:36.284133 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" event={"ID":"b25bbdb8-8bc9-41b7-9b37-8d194ef50087","Type":"ContainerStarted","Data":"0d6bb4632ec2c4e04bd746e9224a3ee63b2f3abf22baf92856d6d794c8efcf9d"} May 26 16:39:37.295322 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:37.295280 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" event={"ID":"b25bbdb8-8bc9-41b7-9b37-8d194ef50087","Type":"ContainerStarted","Data":"2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589"} May 26 16:39:37.296104 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:37.295327 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" event={"ID":"b25bbdb8-8bc9-41b7-9b37-8d194ef50087","Type":"ContainerStarted","Data":"496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8"} May 26 16:39:37.296104 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:37.295484 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:37.319935 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:37.319864 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" podStartSLOduration=3.319841497 podStartE2EDuration="3.319841497s" podCreationTimestamp="2026-05-26 16:39:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:39:37.3168611 +0000 UTC m=+2326.176783565" watchObservedRunningTime="2026-05-26 16:39:37.319841497 +0000 UTC m=+2326.179763963" May 26 16:39:45.170557 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:45.170522 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:45.171166 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:45.170689 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:45.173290 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:45.173268 2573 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:39:45.329848 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:39:45.329810 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:40:07.338531 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:40:07.338502 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:40:52.000046 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:40:52.000014 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:40:52.013205 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:40:52.013182 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:40:53.598179 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:40:53.598146 2573 generic.go:358] "Generic (PLEG): container finished" podID="09b5c3fc-0887-4402-aff2-92600665b90b" containerID="4e58418afcf1e26be6a33aa724f708704c8a9935ce25b9894a4b5a8cefff7f02" exitCode=0 May 26 16:40:53.598540 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:40:53.598215 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"09b5c3fc-0887-4402-aff2-92600665b90b","Type":"ContainerDied","Data":"4e58418afcf1e26be6a33aa724f708704c8a9935ce25b9894a4b5a8cefff7f02"} May 26 16:41:35.329338 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:35.329275 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2"] May 26 16:41:35.330427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:35.330384 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="tokenizer" containerID="cri-o://2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589" gracePeriod=30 May 26 16:41:35.330427 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:35.330321 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="main" containerID="cri-o://496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8" gracePeriod=30 May 26 16:41:35.790187 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:35.790145 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"09b5c3fc-0887-4402-aff2-92600665b90b","Type":"ContainerStarted","Data":"b61945b76e04d69eed858af08450f85195cc58933eb546a13d071b56f635ca68"} May 26 16:41:35.790529 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:35.790499 2573 kubelet_pods.go:1019] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-2m5pn\" not found" May 26 16:41:35.792343 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:35.792314 2573 generic.go:358] "Generic (PLEG): container finished" podID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerID="496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8" exitCode=0 May 26 16:41:35.792470 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:35.792376 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" event={"ID":"b25bbdb8-8bc9-41b7-9b37-8d194ef50087","Type":"ContainerDied","Data":"496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8"} May 26 16:41:35.814360 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:35.814313 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podStartSLOduration=80.233907387 podStartE2EDuration="2m1.814297182s" podCreationTimestamp="2026-05-26 16:39:34 +0000 UTC" firstStartedPulling="2026-05-26 16:40:53.599342317 +0000 UTC m=+2402.459264765" lastFinishedPulling="2026-05-26 16:41:35.179732112 +0000 UTC m=+2444.039654560" observedRunningTime="2026-05-26 16:41:35.812567401 +0000 UTC m=+2444.672489870" watchObservedRunningTime="2026-05-26 16:41:35.814297182 +0000 UTC m=+2444.674219650" May 26 16:41:35.932101 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:35.932062 2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs: secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:35.932284 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:35.932158 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs podName:09b5c3fc-0887-4402-aff2-92600665b90b nodeName:}" failed. No retries permitted until 2026-05-26 16:41:36.432136707 +0000 UTC m=+2445.292059167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs") pod "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b") : secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:36.436930 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:36.436903 2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs: secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:36.437320 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:36.436995 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs podName:09b5c3fc-0887-4402-aff2-92600665b90b nodeName:}" failed. No retries permitted until 2026-05-26 16:41:37.436973312 +0000 UTC m=+2446.296895777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs") pod "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b") : secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:36.622569 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.622543 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:41:36.739996 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.739906 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8tvk\" (UniqueName: \"kubernetes.io/projected/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kube-api-access-k8tvk\") pod \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " May 26 16:41:36.739996 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.739967 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-cache\") pod \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " May 26 16:41:36.740212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.739999 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kserve-provision-location\") pod \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " May 26 16:41:36.740212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.740110 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tls-certs\") pod \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " May 26 16:41:36.740212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.740155 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-tmp\") pod \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " May 26 16:41:36.740212 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.740190 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-uds\") pod \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\" (UID: \"b25bbdb8-8bc9-41b7-9b37-8d194ef50087\") " May 26 16:41:36.740430 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.740231 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b25bbdb8-8bc9-41b7-9b37-8d194ef50087" (UID: "b25bbdb8-8bc9-41b7-9b37-8d194ef50087"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:41:36.740490 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.740474 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:41:36.740544 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.740476 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b25bbdb8-8bc9-41b7-9b37-8d194ef50087" (UID: "b25bbdb8-8bc9-41b7-9b37-8d194ef50087"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:41:36.740544 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.740498 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "b25bbdb8-8bc9-41b7-9b37-8d194ef50087" (UID: "b25bbdb8-8bc9-41b7-9b37-8d194ef50087"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:41:36.740975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.740949 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b25bbdb8-8bc9-41b7-9b37-8d194ef50087" (UID: "b25bbdb8-8bc9-41b7-9b37-8d194ef50087"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:41:36.742171 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.742151 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kube-api-access-k8tvk" (OuterVolumeSpecName: "kube-api-access-k8tvk") pod "b25bbdb8-8bc9-41b7-9b37-8d194ef50087" (UID: "b25bbdb8-8bc9-41b7-9b37-8d194ef50087"). InnerVolumeSpecName "kube-api-access-k8tvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:41:36.742237 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.742187 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b25bbdb8-8bc9-41b7-9b37-8d194ef50087" (UID: "b25bbdb8-8bc9-41b7-9b37-8d194ef50087"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:41:36.799259 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.799223 2573 generic.go:358] "Generic (PLEG): container finished" podID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerID="2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589" exitCode=0 May 26 16:41:36.799443 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.799302 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" event={"ID":"b25bbdb8-8bc9-41b7-9b37-8d194ef50087","Type":"ContainerDied","Data":"2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589"} May 26 16:41:36.799443 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.799338 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" event={"ID":"b25bbdb8-8bc9-41b7-9b37-8d194ef50087","Type":"ContainerDied","Data":"0d6bb4632ec2c4e04bd746e9224a3ee63b2f3abf22baf92856d6d794c8efcf9d"} May 26 16:41:36.799443 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.799357 2573 scope.go:117] "RemoveContainer" containerID="2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589" May 26 16:41:36.799443 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.799309 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2" May 26 16:41:36.799774 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.799754 2573 kubelet_pods.go:1019] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-2m5pn\" not found" May 26 16:41:36.809009 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.808992 2573 scope.go:117] "RemoveContainer" containerID="496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8" May 26 16:41:36.820186 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.820157 2573 scope.go:117] "RemoveContainer" containerID="a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7" May 26 16:41:36.823062 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.823037 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2"] May 26 16:41:36.829211 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.828710 2573 scope.go:117] "RemoveContainer" containerID="2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589" May 26 16:41:36.829324 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:36.829299 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589\": container with ID starting with 2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589 not found: ID does not exist" containerID="2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589" May 26 16:41:36.829396 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.829329 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589"} err="failed to get container status \"2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589\": rpc error: code = NotFound desc = could not find container \"2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589\": container with ID starting with 2455700db9d2481b448c7cc49499aa5f210c914218586817ee099f39576ba589 not found: ID does not exist" May 26 16:41:36.829396 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.829357 2573 scope.go:117] "RemoveContainer" containerID="496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8" May 26 16:41:36.829684 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:36.829648 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8\": container with ID starting with 496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8 not found: ID does not exist" containerID="496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8" May 26 16:41:36.829800 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.829689 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8"} err="failed to get container status \"496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8\": rpc error: code = NotFound desc = could not find container \"496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8\": container with ID starting with 496e6bd32bca324a5548b8340adc7eeb680b324ebca6b4168309b3ab23516fb8 not found: ID does not exist" May 26 16:41:36.829800 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.829715 2573 scope.go:117] "RemoveContainer" containerID="a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7" May 26 16:41:36.830087 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:36.829997 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7\": container with ID starting with a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7 not found: ID does not exist" containerID="a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7" May 26 16:41:36.830087 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.830023 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7"} err="failed to get container status \"a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7\": rpc error: code = NotFound desc = could not find container \"a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7\": container with ID starting with a456f7c6e94f52660c18cff244ea099144935eacaa1a280daac1a9de036ea2d7 not found: ID does not exist" May 26 16:41:36.832087 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.832064 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches4mg2"] May 26 16:41:36.841572 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.841548 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8tvk\" (UniqueName: \"kubernetes.io/projected/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kube-api-access-k8tvk\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:41:36.841572 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.841569 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:41:36.841745 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.841581 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:41:36.841745 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.841592 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-tmp\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:41:36.841745 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:36.841601 2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b25bbdb8-8bc9-41b7-9b37-8d194ef50087-tokenizer-uds\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:41:37.066422 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:37.066387 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] May 26 16:41:37.447905 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:37.447808 2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs: secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:37.447905 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:37.447898 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs podName:09b5c3fc-0887-4402-aff2-92600665b90b nodeName:}" failed. No retries permitted until 2026-05-26 16:41:39.44788249 +0000 UTC m=+2448.307804937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs") pod "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b") : secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:37.804296 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:37.804259 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podUID="09b5c3fc-0887-4402-aff2-92600665b90b" containerName="main" containerID="cri-o://b61945b76e04d69eed858af08450f85195cc58933eb546a13d071b56f635ca68" gracePeriod=30 May 26 16:41:37.874083 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:41:37.874043 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" path="/var/lib/kubelet/pods/b25bbdb8-8bc9-41b7-9b37-8d194ef50087/volumes" May 26 16:41:39.469635 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:39.469601 2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs: secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:39.470126 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:39.469680 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs podName:09b5c3fc-0887-4402-aff2-92600665b90b nodeName:}" failed. No retries permitted until 2026-05-26 16:41:43.469660778 +0000 UTC m=+2452.329583229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs") pod "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b") : secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:43.508721 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:43.508684 2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs: secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:43.509350 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:43.508762 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs podName:09b5c3fc-0887-4402-aff2-92600665b90b nodeName:}" failed. No retries permitted until 2026-05-26 16:41:51.508745491 +0000 UTC m=+2460.368667938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs") pod "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b") : secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:51.597374 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:51.597341 2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs: secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:41:51.597780 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:41:51.597421 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs podName:09b5c3fc-0887-4402-aff2-92600665b90b nodeName:}" failed. No retries permitted until 2026-05-26 16:42:07.597406291 +0000 UTC m=+2476.457328738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs") pod "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b") : secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:42:07.661308 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:42:07.661271 2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs: secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:42:07.661755 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:42:07.661344 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs podName:09b5c3fc-0887-4402-aff2-92600665b90b nodeName:}" failed. No retries permitted until 2026-05-26 16:42:39.661328964 +0000 UTC m=+2508.521251411 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs") pod "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b") : secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found May 26 16:42:07.926674 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:07.926614 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_09b5c3fc-0887-4402-aff2-92600665b90b/main/0.log" May 26 16:42:07.926990 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:07.926967 2573 generic.go:358] "Generic (PLEG): container finished" podID="09b5c3fc-0887-4402-aff2-92600665b90b" containerID="b61945b76e04d69eed858af08450f85195cc58933eb546a13d071b56f635ca68" exitCode=137 May 26 16:42:07.927061 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:07.927042 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"09b5c3fc-0887-4402-aff2-92600665b90b","Type":"ContainerDied","Data":"b61945b76e04d69eed858af08450f85195cc58933eb546a13d071b56f635ca68"} May 26 16:42:08.485187 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.485164 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_09b5c3fc-0887-4402-aff2-92600665b90b/main/0.log" May 26 16:42:08.485543 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.485521 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:42:08.570241 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570201 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-dshm\") pod \"09b5c3fc-0887-4402-aff2-92600665b90b\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " May 26 16:42:08.570439 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570328 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-kserve-provision-location\") pod \"09b5c3fc-0887-4402-aff2-92600665b90b\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " May 26 16:42:08.570439 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570365 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-model-cache\") pod \"09b5c3fc-0887-4402-aff2-92600665b90b\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " May 26 16:42:08.570439 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570402 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nplj\" (UniqueName: \"kubernetes.io/projected/09b5c3fc-0887-4402-aff2-92600665b90b-kube-api-access-7nplj\") pod \"09b5c3fc-0887-4402-aff2-92600665b90b\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " May 26 16:42:08.570613 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570446 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-tmp-dir\") pod \"09b5c3fc-0887-4402-aff2-92600665b90b\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " May 26 16:42:08.570613 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570505 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs\") pod \"09b5c3fc-0887-4402-aff2-92600665b90b\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " May 26 16:42:08.570613 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570549 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-home\") pod \"09b5c3fc-0887-4402-aff2-92600665b90b\" (UID: \"09b5c3fc-0887-4402-aff2-92600665b90b\") " May 26 16:42:08.570613 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570593 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-model-cache" (OuterVolumeSpecName: "model-cache") pod "09b5c3fc-0887-4402-aff2-92600665b90b" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:42:08.570924 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570890 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09b5c3fc-0887-4402-aff2-92600665b90b" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:42:08.571079 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.570959 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-model-cache\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:42:08.571079 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.571020 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-home" (OuterVolumeSpecName: "home") pod "09b5c3fc-0887-4402-aff2-92600665b90b" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:42:08.572459 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.572429 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-dshm" (OuterVolumeSpecName: "dshm") pod "09b5c3fc-0887-4402-aff2-92600665b90b" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:42:08.572585 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.572564 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09b5c3fc-0887-4402-aff2-92600665b90b" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 26 16:42:08.572647 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.572605 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b5c3fc-0887-4402-aff2-92600665b90b-kube-api-access-7nplj" (OuterVolumeSpecName: "kube-api-access-7nplj") pod "09b5c3fc-0887-4402-aff2-92600665b90b" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b"). InnerVolumeSpecName "kube-api-access-7nplj". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:42:08.625594 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.625513 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09b5c3fc-0887-4402-aff2-92600665b90b" (UID: "09b5c3fc-0887-4402-aff2-92600665b90b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:42:08.672001 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.671968 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b5c3fc-0887-4402-aff2-92600665b90b-tls-certs\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:42:08.672001 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.671996 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-home\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:42:08.672001 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.672007 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-dshm\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:42:08.672426 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.672016 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-kserve-provision-location\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:42:08.672426 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.672026 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nplj\" (UniqueName: \"kubernetes.io/projected/09b5c3fc-0887-4402-aff2-92600665b90b-kube-api-access-7nplj\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:42:08.672426 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.672034 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b5c3fc-0887-4402-aff2-92600665b90b-tmp-dir\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:42:08.931767 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.931685 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_09b5c3fc-0887-4402-aff2-92600665b90b/main/0.log" May 26 16:42:08.932083 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.932060 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"09b5c3fc-0887-4402-aff2-92600665b90b","Type":"ContainerDied","Data":"a698bc16758807e2fe4cab0bcdfebdc1767c58e699c750aa8043f2f7f923363c"} May 26 16:42:08.932189 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.932103 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" May 26 16:42:08.932189 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.932104 2573 scope.go:117] "RemoveContainer" containerID="b61945b76e04d69eed858af08450f85195cc58933eb546a13d071b56f635ca68" May 26 16:42:08.941462 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.941447 2573 scope.go:117] "RemoveContainer" containerID="4e58418afcf1e26be6a33aa724f708704c8a9935ce25b9894a4b5a8cefff7f02" May 26 16:42:08.955520 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.955485 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] May 26 16:42:08.961871 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:08.961850 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] May 26 16:42:09.874272 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:42:09.874241 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09b5c3fc-0887-4402-aff2-92600665b90b" path="/var/lib/kubelet/pods/09b5c3fc-0887-4402-aff2-92600665b90b/volumes" May 26 16:43:06.850401 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850365 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-snwjk/must-gather-8vtrn"] May 26 16:43:06.850845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850757 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="storage-initializer" May 26 16:43:06.850845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850770 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="storage-initializer" May 26 16:43:06.850845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850791 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b5c3fc-0887-4402-aff2-92600665b90b" containerName="storage-initializer" May 26 16:43:06.850845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850797 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b5c3fc-0887-4402-aff2-92600665b90b" containerName="storage-initializer" May 26 16:43:06.850845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850805 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="main" May 26 16:43:06.850845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850810 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="main" May 26 16:43:06.850845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850816 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="tokenizer" May 26 16:43:06.850845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850821 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="tokenizer" May 26 16:43:06.850845 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850845 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b5c3fc-0887-4402-aff2-92600665b90b" containerName="main" May 26 16:43:06.851131 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850851 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b5c3fc-0887-4402-aff2-92600665b90b" containerName="main" May 26 16:43:06.851131 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850913 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="09b5c3fc-0887-4402-aff2-92600665b90b" containerName="main" May 26 16:43:06.851131 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850922 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="main" May 26 16:43:06.851131 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.850929 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="b25bbdb8-8bc9-41b7-9b37-8d194ef50087" containerName="tokenizer" May 26 16:43:06.854058 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.854039 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:06.856645 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.856615 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-snwjk\"/\"openshift-service-ca.crt\"" May 26 16:43:06.856645 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.856639 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-snwjk\"/\"default-dockercfg-ssnvw\"" May 26 16:43:06.857409 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.857392 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-snwjk\"/\"kube-root-ca.crt\"" May 26 16:43:06.863040 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:06.863018 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-snwjk/must-gather-8vtrn"] May 26 16:43:07.007490 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:07.007431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv85z\" (UniqueName: \"kubernetes.io/projected/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-kube-api-access-lv85z\") pod \"must-gather-8vtrn\" (UID: \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\") " pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:07.007706 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:07.007595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-must-gather-output\") pod \"must-gather-8vtrn\" (UID: \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\") " pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:07.108590 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:07.108487 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lv85z\" (UniqueName: \"kubernetes.io/projected/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-kube-api-access-lv85z\") pod \"must-gather-8vtrn\" (UID: \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\") " pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:07.108590 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:07.108580 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-must-gather-output\") pod \"must-gather-8vtrn\" (UID: \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\") " pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:07.108961 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:07.108941 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-must-gather-output\") pod \"must-gather-8vtrn\" (UID: \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\") " pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:07.120701 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:07.120677 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv85z\" (UniqueName: \"kubernetes.io/projected/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-kube-api-access-lv85z\") pod \"must-gather-8vtrn\" (UID: \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\") " pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:07.164736 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:07.164701 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:07.300595 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:07.300564 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-snwjk/must-gather-8vtrn"] May 26 16:43:07.302366 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:43:07.302336 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8a4a638_9c2b_4f37_be9b_22bfed46fd38.slice/crio-9cb5bbf7ada0ebf6cccd8b39a4dea63a4ad02f140e7a6c296c711aa4c36039ad WatchSource:0}: Error finding container 9cb5bbf7ada0ebf6cccd8b39a4dea63a4ad02f140e7a6c296c711aa4c36039ad: Status 404 returned error can't find the container with id 9cb5bbf7ada0ebf6cccd8b39a4dea63a4ad02f140e7a6c296c711aa4c36039ad May 26 16:43:08.181066 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:08.181022 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-snwjk/must-gather-8vtrn" event={"ID":"a8a4a638-9c2b-4f37-be9b-22bfed46fd38","Type":"ContainerStarted","Data":"9cb5bbf7ada0ebf6cccd8b39a4dea63a4ad02f140e7a6c296c711aa4c36039ad"} May 26 16:43:14.208918 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:14.208876 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-snwjk/must-gather-8vtrn" event={"ID":"a8a4a638-9c2b-4f37-be9b-22bfed46fd38","Type":"ContainerStarted","Data":"f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97"} May 26 16:43:14.208918 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:14.208915 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-snwjk/must-gather-8vtrn" event={"ID":"a8a4a638-9c2b-4f37-be9b-22bfed46fd38","Type":"ContainerStarted","Data":"fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38"} May 26 16:43:14.226421 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:14.226362 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-snwjk/must-gather-8vtrn" podStartSLOduration=2.10558739 podStartE2EDuration="8.226345298s" podCreationTimestamp="2026-05-26 16:43:06 +0000 UTC" firstStartedPulling="2026-05-26 16:43:07.303994946 +0000 UTC m=+2536.163917393" lastFinishedPulling="2026-05-26 16:43:13.42475285 +0000 UTC m=+2542.284675301" observedRunningTime="2026-05-26 16:43:14.225093208 +0000 UTC m=+2543.085015676" watchObservedRunningTime="2026-05-26 16:43:14.226345298 +0000 UTC m=+2543.086267767" May 26 16:43:22.777371 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:22.777339 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:23.748567 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:23.748538 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:24.711686 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:24.711657 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:25.665025 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:25.664993 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:26.609969 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:26.609941 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:27.554844 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:27.554805 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:28.505729 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:28.505702 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:29.438784 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:29.438749 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:30.409265 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:30.409219 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:31.349946 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:31.349919 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:32.314666 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:32.314630 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:33.269051 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:33.269022 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:34.265089 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:34.265058 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:35.276023 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:35.275995 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-rj2lf_e17698f4-89e1-4876-babd-e3f49cd03eba/istio-proxy/0.log" May 26 16:43:38.144985 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:38.144958 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-2xmqp_b9e67ba5-1cae-4c39-8e2a-4290d13495cb/manager/0.log" May 26 16:43:38.162574 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:38.162545 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-vnsbs_ec1e07d8-5bdf-4566-a249-24a44c0eb79e/kuadrant-console-plugin/0.log" May 26 16:43:38.250858 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:38.250818 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-j5jj5_c3d67fbb-b733-4b66-ba79-a3e9aa36e51d/limitador/0.log" May 26 16:43:39.321049 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:39.321017 2573 generic.go:358] "Generic (PLEG): container finished" podID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" containerID="fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38" exitCode=0 May 26 16:43:39.321482 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:39.321091 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-snwjk/must-gather-8vtrn" event={"ID":"a8a4a638-9c2b-4f37-be9b-22bfed46fd38","Type":"ContainerDied","Data":"fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38"} May 26 16:43:39.321482 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:39.321421 2573 scope.go:117] "RemoveContainer" containerID="fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38" May 26 16:43:39.999076 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:39.999049 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-snwjk_must-gather-8vtrn_a8a4a638-9c2b-4f37-be9b-22bfed46fd38/gather/0.log" May 26 16:43:43.485891 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:43.485861 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gjfr7_89ec24bc-600b-4510-a6e5-95c672ab9577/global-pull-secret-syncer/0.log" May 26 16:43:43.554669 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:43.554639 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7m96n_616802f2-6a23-4737-921e-862e05ea782c/konnectivity-agent/0.log" May 26 16:43:43.710712 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:43.710681 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-48.ec2.internal_5f6d5d2a73b24fd057a0b80db1b7899f/haproxy/0.log" May 26 16:43:45.477988 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.477951 2573 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-snwjk/must-gather-8vtrn"] May 26 16:43:45.478450 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.478229 2573 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-snwjk/must-gather-8vtrn" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" containerName="copy" containerID="cri-o://f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97" gracePeriod=2 May 26 16:43:45.480653 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.480610 2573 status_manager.go:895] "Failed to get status for pod" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" pod="openshift-must-gather-snwjk/must-gather-8vtrn" err="pods \"must-gather-8vtrn\" is forbidden: User \"system:node:ip-10-0-140-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-snwjk\": no relationship found between node 'ip-10-0-140-48.ec2.internal' and this object" May 26 16:43:45.483038 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.483014 2573 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-snwjk/must-gather-8vtrn"] May 26 16:43:45.715991 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.715965 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-snwjk_must-gather-8vtrn_a8a4a638-9c2b-4f37-be9b-22bfed46fd38/copy/0.log" May 26 16:43:45.716334 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.716318 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:45.720589 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.720565 2573 status_manager.go:895] "Failed to get status for pod" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" pod="openshift-must-gather-snwjk/must-gather-8vtrn" err="pods \"must-gather-8vtrn\" is forbidden: User \"system:node:ip-10-0-140-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-snwjk\": no relationship found between node 'ip-10-0-140-48.ec2.internal' and this object" May 26 16:43:45.873634 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.873601 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-must-gather-output\") pod \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\" (UID: \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\") " May 26 16:43:45.873818 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.873718 2573 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv85z\" (UniqueName: \"kubernetes.io/projected/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-kube-api-access-lv85z\") pod \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\" (UID: \"a8a4a638-9c2b-4f37-be9b-22bfed46fd38\") " May 26 16:43:45.876006 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.875982 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-kube-api-access-lv85z" (OuterVolumeSpecName: "kube-api-access-lv85z") pod "a8a4a638-9c2b-4f37-be9b-22bfed46fd38" (UID: "a8a4a638-9c2b-4f37-be9b-22bfed46fd38"). InnerVolumeSpecName "kube-api-access-lv85z". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 26 16:43:45.879021 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.878998 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a8a4a638-9c2b-4f37-be9b-22bfed46fd38" (UID: "a8a4a638-9c2b-4f37-be9b-22bfed46fd38"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 26 16:43:45.975469 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.975434 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv85z\" (UniqueName: \"kubernetes.io/projected/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-kube-api-access-lv85z\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:43:45.975662 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:45.975481 2573 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a8a4a638-9c2b-4f37-be9b-22bfed46fd38-must-gather-output\") on node \"ip-10-0-140-48.ec2.internal\" DevicePath \"\"" May 26 16:43:46.355970 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:46.355942 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-snwjk_must-gather-8vtrn_a8a4a638-9c2b-4f37-be9b-22bfed46fd38/copy/0.log" May 26 16:43:46.356311 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:46.356289 2573 generic.go:358] "Generic (PLEG): container finished" podID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" containerID="f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97" exitCode=143 May 26 16:43:46.356380 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:46.356340 2573 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-snwjk/must-gather-8vtrn" May 26 16:43:46.356434 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:46.356389 2573 scope.go:117] "RemoveContainer" containerID="f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97" May 26 16:43:46.365672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:46.365655 2573 scope.go:117] "RemoveContainer" containerID="fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38" May 26 16:43:46.379403 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:46.379383 2573 scope.go:117] "RemoveContainer" containerID="f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97" May 26 16:43:46.379629 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:43:46.379610 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97\": container with ID starting with f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97 not found: ID does not exist" containerID="f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97" May 26 16:43:46.379708 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:46.379642 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97"} err="failed to get container status \"f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97\": rpc error: code = NotFound desc = could not find container \"f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97\": container with ID starting with f92ca7173206a515ef610369c09f9a5dcc58928d9e985f9658bb6d6b64d83d97 not found: ID does not exist" May 26 16:43:46.379708 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:46.379668 2573 scope.go:117] "RemoveContainer" containerID="fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38" May 26 16:43:46.380005 ip-10-0-140-48 kubenswrapper[2573]: E0526 16:43:46.379987 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38\": container with ID starting with fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38 not found: ID does not exist" containerID="fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38" May 26 16:43:46.380055 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:46.380012 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38"} err="failed to get container status \"fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38\": rpc error: code = NotFound desc = could not find container \"fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38\": container with ID starting with fc818f310135981c8cb8d4dd65a492a7b28844c9ed3def8a57d86cb1afd77c38 not found: ID does not exist" May 26 16:43:47.874672 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:47.874636 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" path="/var/lib/kubelet/pods/a8a4a638-9c2b-4f37-be9b-22bfed46fd38/volumes" May 26 16:43:48.048542 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:48.048511 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-2xmqp_b9e67ba5-1cae-4c39-8e2a-4290d13495cb/manager/0.log" May 26 16:43:48.086896 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:48.086869 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-vnsbs_ec1e07d8-5bdf-4566-a249-24a44c0eb79e/kuadrant-console-plugin/0.log" May 26 16:43:48.172287 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:48.172207 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-j5jj5_c3d67fbb-b733-4b66-ba79-a3e9aa36e51d/limitador/0.log" May 26 16:43:49.196204 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.196175 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bb6bbc86-422e-4657-8ef7-0351f07efa06/alertmanager/0.log" May 26 16:43:49.219710 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.219669 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bb6bbc86-422e-4657-8ef7-0351f07efa06/config-reloader/0.log" May 26 16:43:49.241897 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.241873 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bb6bbc86-422e-4657-8ef7-0351f07efa06/kube-rbac-proxy-web/0.log" May 26 16:43:49.265122 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.265095 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bb6bbc86-422e-4657-8ef7-0351f07efa06/kube-rbac-proxy/0.log" May 26 16:43:49.295347 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.295310 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bb6bbc86-422e-4657-8ef7-0351f07efa06/kube-rbac-proxy-metric/0.log" May 26 16:43:49.323456 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.323431 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bb6bbc86-422e-4657-8ef7-0351f07efa06/prom-label-proxy/0.log" May 26 16:43:49.356120 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.356093 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bb6bbc86-422e-4657-8ef7-0351f07efa06/init-config-reloader/0.log" May 26 16:43:49.573221 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.573176 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-kqb24_0e3211cf-6726-4e08-b6be-a85fbd33eb2c/monitoring-plugin/0.log" May 26 16:43:49.758942 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.758912 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dctfv_aceee1e1-ee32-492f-a009-d97fb515b672/node-exporter/0.log" May 26 16:43:49.790028 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.789991 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dctfv_aceee1e1-ee32-492f-a009-d97fb515b672/kube-rbac-proxy/0.log" May 26 16:43:49.818228 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.818201 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dctfv_aceee1e1-ee32-492f-a009-d97fb515b672/init-textfile/0.log" May 26 16:43:49.952156 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.952060 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-4qch8_a315e186-fcad-4ae6-adf5-74b3a6e5903b/kube-rbac-proxy-main/0.log" May 26 16:43:49.976769 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:49.976738 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-4qch8_a315e186-fcad-4ae6-adf5-74b3a6e5903b/kube-rbac-proxy-self/0.log" May 26 16:43:50.004361 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.004323 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-4qch8_a315e186-fcad-4ae6-adf5-74b3a6e5903b/openshift-state-metrics/0.log" May 26 16:43:50.248663 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.248585 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-42lfv_3e72ca05-4dae-4954-b79c-f9c34a00ed13/prometheus-operator/0.log" May 26 16:43:50.279186 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.279160 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-42lfv_3e72ca05-4dae-4954-b79c-f9c34a00ed13/kube-rbac-proxy/0.log" May 26 16:43:50.315598 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.315568 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-k6ndj_eb6302e4-7513-475a-97a2-62fcbc91f37c/prometheus-operator-admission-webhook/0.log" May 26 16:43:50.349540 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.349517 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8548648857-6pkmp_0aa5eec1-ca38-4545-a077-e6c873b4c7c5/telemeter-client/0.log" May 26 16:43:50.379572 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.379545 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8548648857-6pkmp_0aa5eec1-ca38-4545-a077-e6c873b4c7c5/reload/0.log" May 26 16:43:50.406262 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.406221 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8548648857-6pkmp_0aa5eec1-ca38-4545-a077-e6c873b4c7c5/kube-rbac-proxy/0.log" May 26 16:43:50.443018 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.442994 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c47c44689-7g8p2_04523e11-5c21-4b16-aea5-ba8ed8999e39/thanos-query/0.log" May 26 16:43:50.476663 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.476625 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c47c44689-7g8p2_04523e11-5c21-4b16-aea5-ba8ed8999e39/kube-rbac-proxy-web/0.log" May 26 16:43:50.508247 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.508164 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c47c44689-7g8p2_04523e11-5c21-4b16-aea5-ba8ed8999e39/kube-rbac-proxy/0.log" May 26 16:43:50.562627 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.562599 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c47c44689-7g8p2_04523e11-5c21-4b16-aea5-ba8ed8999e39/prom-label-proxy/0.log" May 26 16:43:50.593229 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.593187 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c47c44689-7g8p2_04523e11-5c21-4b16-aea5-ba8ed8999e39/kube-rbac-proxy-rules/0.log" May 26 16:43:50.622636 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:50.622610 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c47c44689-7g8p2_04523e11-5c21-4b16-aea5-ba8ed8999e39/kube-rbac-proxy-metrics/0.log" May 26 16:43:52.264634 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.264579 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/1.log" May 26 16:43:52.269465 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.269439 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-psh92_a782d182-009d-4d16-babd-26586d23986b/console-operator/2.log" May 26 16:43:52.522922 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.522811 2573 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6"] May 26 16:43:52.523343 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.523322 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" containerName="copy" May 26 16:43:52.523428 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.523345 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" containerName="copy" May 26 16:43:52.523428 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.523360 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" containerName="gather" May 26 16:43:52.523428 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.523368 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" containerName="gather" May 26 16:43:52.523600 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.523454 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" containerName="copy" May 26 16:43:52.523600 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.523466 2573 memory_manager.go:356] "RemoveStaleState removing state" podUID="a8a4a638-9c2b-4f37-be9b-22bfed46fd38" containerName="gather" May 26 16:43:52.528102 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.528077 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.531624 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.531598 2573 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9f8jk\"/\"default-dockercfg-bg6lc\"" May 26 16:43:52.531754 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.531644 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f8jk\"/\"openshift-service-ca.crt\"" May 26 16:43:52.532320 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.532306 2573 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f8jk\"/\"kube-root-ca.crt\"" May 26 16:43:52.532687 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.532666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2p8n\" (UniqueName: \"kubernetes.io/projected/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-kube-api-access-g2p8n\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.532768 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.532698 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-lib-modules\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.532806 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.532773 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-proc\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.532916 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.532903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-sys\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.532975 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.532942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-podres\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.538021 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.538000 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6"] May 26 16:43:52.633667 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.633634 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-sys\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.633667 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.633680 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-podres\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.633947 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.633702 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2p8n\" (UniqueName: \"kubernetes.io/projected/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-kube-api-access-g2p8n\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.633947 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.633720 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-lib-modules\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.633947 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.633741 2573 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-proc\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.633947 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.633752 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-sys\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.633947 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.633823 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-proc\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.633947 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.633897 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-podres\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.633947 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.633901 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-lib-modules\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.642799 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.642783 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2p8n\" (UniqueName: \"kubernetes.io/projected/f9cf4e93-e666-4ade-9a6c-33f5e256d58d-kube-api-access-g2p8n\") pod \"perf-node-gather-daemonset-bpnl6\" (UID: \"f9cf4e93-e666-4ade-9a6c-33f5e256d58d\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.792688 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.792612 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-988cdc694-h4mq5_6a24e1b4-cb4e-444f-838a-91327757542b/console/0.log" May 26 16:43:52.825578 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.825546 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-pt9k8_6b080ee4-6ec6-4729-a3dd-7ae4473f41b1/download-server/0.log" May 26 16:43:52.839944 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.839910 2573 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:52.971983 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:52.971951 2573 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6"] May 26 16:43:52.973136 ip-10-0-140-48 kubenswrapper[2573]: W0526 16:43:52.973112 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9cf4e93_e666_4ade_9a6c_33f5e256d58d.slice/crio-aa2113304525dff280326e989bc6dcec5e1d49d1e1e434901f7e9cb1b45ff5f0 WatchSource:0}: Error finding container aa2113304525dff280326e989bc6dcec5e1d49d1e1e434901f7e9cb1b45ff5f0: Status 404 returned error can't find the container with id aa2113304525dff280326e989bc6dcec5e1d49d1e1e434901f7e9cb1b45ff5f0 May 26 16:43:53.311763 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:53.311682 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-flgsz_27f5c521-61a5-4184-855b-243fd8577783/volume-data-source-validator/0.log" May 26 16:43:53.384242 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:53.384200 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" event={"ID":"f9cf4e93-e666-4ade-9a6c-33f5e256d58d","Type":"ContainerStarted","Data":"4f0b00b425172da33e87ee4a3d9ae84b8011f0179e05d0e896894befb7ee1f62"} May 26 16:43:53.384242 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:53.384235 2573 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" event={"ID":"f9cf4e93-e666-4ade-9a6c-33f5e256d58d","Type":"ContainerStarted","Data":"aa2113304525dff280326e989bc6dcec5e1d49d1e1e434901f7e9cb1b45ff5f0"} May 26 16:43:53.384456 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:53.384332 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:53.405374 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:53.405308 2573 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" podStartSLOduration=1.405292249 podStartE2EDuration="1.405292249s" podCreationTimestamp="2026-05-26 16:43:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:43:53.404082492 +0000 UTC m=+2582.264004961" watchObservedRunningTime="2026-05-26 16:43:53.405292249 +0000 UTC m=+2582.265214746" May 26 16:43:54.233137 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:54.233105 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p8lq6_c42e8c71-312c-41a0-a8d9-b116879302ae/dns/0.log" May 26 16:43:54.257204 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:54.257172 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p8lq6_c42e8c71-312c-41a0-a8d9-b116879302ae/kube-rbac-proxy/0.log" May 26 16:43:54.332467 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:54.332439 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjm54_e636febb-62af-4bae-87f2-f5ba6ee79f71/dns-node-resolver/0.log" May 26 16:43:54.876039 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:54.876010 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b4gnl_5543c7e9-d0b5-444a-94bf-cf6d6bdf7f9a/node-ca/0.log" May 26 16:43:56.409181 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:56.409144 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rs5nq_ce3ca0a4-5dbd-425b-a0d3-3f2688605a3a/serve-healthcheck-canary/0.log" May 26 16:43:56.881561 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:56.881534 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-tbzzs_317da9eb-b20e-413a-af49-fde829ee35d5/insights-operator/0.log" May 26 16:43:56.882125 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:56.882108 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-tbzzs_317da9eb-b20e-413a-af49-fde829ee35d5/insights-operator/1.log" May 26 16:43:57.070165 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:57.070129 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hsw96_dbf34c33-386f-4e41-a50d-b543f527e1ce/kube-rbac-proxy/0.log" May 26 16:43:57.098611 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:57.098584 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hsw96_dbf34c33-386f-4e41-a50d-b543f527e1ce/exporter/0.log" May 26 16:43:57.122707 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:57.122684 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hsw96_dbf34c33-386f-4e41-a50d-b543f527e1ce/extractor/0.log" May 26 16:43:59.398795 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:59.398767 2573 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-bpnl6" May 26 16:43:59.740716 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:59.740632 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-669cc6566-wwl29_af570fde-da98-443c-9544-668d15604078/manager/0.log" May 26 16:43:59.770549 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:43:59.770519 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-bfc7f696d-j5hlb_0f660637-c4c2-42fa-ada9-c8b59e42b5a1/openshift-lws-operator/0.log" May 26 16:44:00.683792 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:00.683760 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-spwx2_cd6a3428-c45c-46e4-9d1f-b10d1c74a4b6/s3-init/0.log" May 26 16:44:05.414029 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:05.413996 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-lq98j_51494e0f-df60-46c9-859c-4bb91e58078b/migrator/0.log" May 26 16:44:05.437547 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:05.437517 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-lq98j_51494e0f-df60-46c9-859c-4bb91e58078b/graceful-termination/0.log" May 26 16:44:05.797542 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:05.797511 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-x8qn9_5ade83d5-dc58-49a9-b12d-0ef025b0d373/kube-storage-version-migrator-operator/1.log" May 26 16:44:05.798361 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:05.798341 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-x8qn9_5ade83d5-dc58-49a9-b12d-0ef025b0d373/kube-storage-version-migrator-operator/0.log" May 26 16:44:06.973309 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:06.973284 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r4lmb_5fee4bfe-ae19-487c-a75b-cf4f2a013075/kube-multus-additional-cni-plugins/0.log" May 26 16:44:06.999035 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:06.998996 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r4lmb_5fee4bfe-ae19-487c-a75b-cf4f2a013075/egress-router-binary-copy/0.log" May 26 16:44:07.025244 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:07.025203 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r4lmb_5fee4bfe-ae19-487c-a75b-cf4f2a013075/cni-plugins/0.log" May 26 16:44:07.048371 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:07.048337 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r4lmb_5fee4bfe-ae19-487c-a75b-cf4f2a013075/bond-cni-plugin/0.log" May 26 16:44:07.074159 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:07.074131 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r4lmb_5fee4bfe-ae19-487c-a75b-cf4f2a013075/routeoverride-cni/0.log" May 26 16:44:07.102105 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:07.102080 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r4lmb_5fee4bfe-ae19-487c-a75b-cf4f2a013075/whereabouts-cni-bincopy/0.log" May 26 16:44:07.130694 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:07.130670 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r4lmb_5fee4bfe-ae19-487c-a75b-cf4f2a013075/whereabouts-cni/0.log" May 26 16:44:07.387487 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:07.387453 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vn7x4_7f746638-ec1d-4cc7-8e72-6f7ef6991c06/kube-multus/0.log" May 26 16:44:07.636438 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:07.636406 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vz84d_264cf2a2-6d0f-4220-8797-88813b9a33fe/network-metrics-daemon/0.log" May 26 16:44:07.656652 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:07.656578 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vz84d_264cf2a2-6d0f-4220-8797-88813b9a33fe/kube-rbac-proxy/0.log" May 26 16:44:08.493658 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:08.493586 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9pgwm_5b420341-5a85-4184-9d56-616ba0fa532c/ovn-controller/0.log" May 26 16:44:08.522564 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:08.522539 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9pgwm_5b420341-5a85-4184-9d56-616ba0fa532c/ovn-acl-logging/0.log" May 26 16:44:08.543371 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:08.543350 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9pgwm_5b420341-5a85-4184-9d56-616ba0fa532c/kube-rbac-proxy-node/0.log" May 26 16:44:08.566590 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:08.566568 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9pgwm_5b420341-5a85-4184-9d56-616ba0fa532c/kube-rbac-proxy-ovn-metrics/0.log" May 26 16:44:08.589284 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:08.589264 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9pgwm_5b420341-5a85-4184-9d56-616ba0fa532c/northd/0.log" May 26 16:44:08.610392 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:08.610372 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9pgwm_5b420341-5a85-4184-9d56-616ba0fa532c/nbdb/0.log" May 26 16:44:08.631521 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:08.631497 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9pgwm_5b420341-5a85-4184-9d56-616ba0fa532c/sbdb/0.log" May 26 16:44:08.761041 ip-10-0-140-48 kubenswrapper[2573]: I0526 16:44:08.760961 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9pgwm_5b420341-5a85-4184-9d56-616ba0fa532c/ovnkube-controller/0.log"