May 28 03:14:08.224960 ip-10-0-141-29 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 28 03:14:08.224972 ip-10-0-141-29 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 28 03:14:08.224982 ip-10-0-141-29 systemd[1]: kubelet.service: Failed with result 'resources'.
May 28 03:14:08.225300 ip-10-0-141-29 systemd[1]: Failed to start Kubernetes Kubelet.
May 28 03:14:18.412382 ip-10-0-141-29 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 28 03:14:18.412400 ip-10-0-141-29 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f0fbf653a9fe4bc5a12cf171848c63dc --
May 28 03:16:48.166689 ip-10-0-141-29 systemd[1]: Starting Kubernetes Kubelet...
May 28 03:16:48.615407 ip-10-0-141-29 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 03:16:48.615407 ip-10-0-141-29 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 28 03:16:48.615407 ip-10-0-141-29 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 03:16:48.615407 ip-10-0-141-29 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 28 03:16:48.615407 ip-10-0-141-29 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 03:16:48.618221 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.618129    2569 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 28 03:16:48.622502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622486    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 03:16:48.622502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622501    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622505    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622508    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622511    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622514    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622517    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622519    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622522    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622525    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622528    2569 feature_gate.go:328] unrecognized feature gate: Example
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622531    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622534    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622537    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622542    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622544    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622547    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622550    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622552    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622555    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 03:16:48.622565 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622558    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622560    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622563    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622566    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622568    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622571    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622574    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622577    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622593    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622596    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622598    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622601    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622603    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622606    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622609    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622611    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622614    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622617    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622620    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622622    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 03:16:48.623007 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622625    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622628    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622631    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622633    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622636    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622638    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622641    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622644    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622646    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622649    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622652    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622655    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622657    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622660    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622662    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622671    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622674    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622677    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622680    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 03:16:48.623502 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622683    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622686    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622688    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622691    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622694    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622696    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622699    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622703    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622706    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622708    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622711    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622713    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622716    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622720    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622723    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622727    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622729    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622732    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622735    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622737    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 03:16:48.624004 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622740    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622742    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622747    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622750    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622753    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622755    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.622758    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623145    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623149    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623152    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623155    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623158    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623161    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623163    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623168    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623172    2569 feature_gate.go:328] unrecognized feature gate: Example
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623176    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623179    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623182    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623185    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 03:16:48.624501 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623188    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623191    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623194    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623197    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623199    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623202    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623205    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623207    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623210    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623213    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623215    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623218    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623220    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623223    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623227    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623229    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623232    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623235    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623238    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623240    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 03:16:48.624969 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623243    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623245    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623249    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623251    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623254    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623256    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623259    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623261    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623264    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623266    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623269    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623271    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623274    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623277    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623279    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623282    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623284    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623287    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623289    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623292    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 03:16:48.625490 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623294    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623297    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623299    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623301    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623304    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623307    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623310    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623313    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623316    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623318    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623321    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623324    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623326    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623329    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623331    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623334    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623351    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623354    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623356    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623359    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 03:16:48.625973 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623362    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623364    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623367    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623369    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623372    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623375    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623377    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623380    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623383    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623387    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623391    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623394    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.623396    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624620    2569 flags.go:64] FLAG: --address="0.0.0.0"
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624630    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624637    2569 flags.go:64] FLAG: --anonymous-auth="true"
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624641    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624646    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624649    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624654    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624659    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 28 03:16:48.626464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624662    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624665    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624669    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624672    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624676    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624679    2569 flags.go:64] FLAG: --cgroup-root=""
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624682    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624685    2569 flags.go:64] FLAG: --client-ca-file=""
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624688    2569 flags.go:64] FLAG: --cloud-config=""
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624691    2569 flags.go:64] FLAG: --cloud-provider="external"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624694    2569 flags.go:64] FLAG: --cluster-dns="[]"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624699    2569 flags.go:64] FLAG: --cluster-domain=""
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624702    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624708    2569 flags.go:64] FLAG: --config-dir=""
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624711    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624714    2569 flags.go:64] FLAG: --container-log-max-files="5"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624718    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624722    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624726    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624729    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624732    2569 flags.go:64] FLAG: --contention-profiling="false"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624735    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624738    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624741    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624744    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 28 03:16:48.626966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624748    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624751    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624754    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624757    2569 flags.go:64] FLAG: --enable-load-reader="false"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624761    2569 flags.go:64] FLAG: --enable-server="true"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624764    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624768    2569 flags.go:64] FLAG: --event-burst="100"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624771    2569 flags.go:64] FLAG: --event-qps="50"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624774    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624778    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624781    2569 flags.go:64] FLAG: --eviction-hard=""
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624784    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624787    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624790    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624794    2569 flags.go:64] FLAG: --eviction-soft=""
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624797    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624799    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624803    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624806    2569 flags.go:64] FLAG: --experimental-mounter-path=""
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624809    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624813    2569 flags.go:64] FLAG: --fail-swap-on="true"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624817    2569 flags.go:64] FLAG: --feature-gates=""
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624821    2569 flags.go:64] FLAG: --file-check-frequency="20s"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624824    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624827    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 28 03:16:48.627670 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624831    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624834    2569 flags.go:64] FLAG: --healthz-port="10248"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624837    2569 flags.go:64] FLAG: --help="false"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624840    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-141-29.ec2.internal"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624844    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624847    2569 flags.go:64] FLAG: --http-check-frequency="20s"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624850    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624854    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624858    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624861    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624864    2569 flags.go:64] FLAG: --image-service-endpoint=""
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624867    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624870    2569 flags.go:64] FLAG: --kube-api-burst="100"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624872    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624876    2569 flags.go:64] FLAG: --kube-api-qps="50"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624879    2569 flags.go:64] FLAG: --kube-reserved=""
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624882    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624885    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624888    2569 flags.go:64] FLAG: --kubelet-cgroups=""
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624891    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624894    2569 flags.go:64] FLAG: --lock-file=""
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624897    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624899    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624902    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 28 03:16:48.628259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624907    2569 flags.go:64] FLAG: --log-json-split-stream="false"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624910    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624914    2569 flags.go:64] FLAG: --log-text-split-stream="false"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624918    2569 flags.go:64] FLAG: --logging-format="text"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624920    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624924    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624927    2569 flags.go:64] FLAG: --manifest-url=""
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624930    2569 flags.go:64] FLAG: --manifest-url-header=""
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624934    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624938    2569 flags.go:64] FLAG: --max-open-files="1000000"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624942    2569 flags.go:64] FLAG: --max-pods="110"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624945    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624948    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624951    2569 flags.go:64] FLAG: --memory-manager-policy="None"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624954    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624958    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624961    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624964    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624987    2569 flags.go:64] FLAG: --node-status-max-images="50"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624990    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624993    2569 flags.go:64] FLAG: --oom-score-adj="-999"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.624996    2569 flags.go:64] FLAG: --pod-cidr=""
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625006    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 28 03:16:48.628885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625012    2569 flags.go:64] FLAG: --pod-manifest-path=""
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625015    2569 flags.go:64] FLAG: --pod-max-pids="-1"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625018    2569 flags.go:64] FLAG: --pods-per-core="0"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625022    2569 flags.go:64] FLAG: --port="10250"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625025    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625028    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0816f8cfa070c1129"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625031    2569 flags.go:64] FLAG: --qos-reserved=""
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625034    2569 flags.go:64] FLAG: --read-only-port="10255"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625037    2569 flags.go:64] FLAG: --register-node="true"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625040    2569 flags.go:64] FLAG: --register-schedulable="true"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625043    2569 flags.go:64] FLAG: --register-with-taints=""
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625055    2569 flags.go:64] FLAG: --registry-burst="10"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625059    2569 flags.go:64] FLAG: --registry-qps="5"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625061    2569 flags.go:64] FLAG: --reserved-cpus=""
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625064    2569 flags.go:64] FLAG: --reserved-memory=""
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625069    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625072    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625075    2569 flags.go:64] FLAG: --rotate-certificates="false"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625078    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625081    2569 flags.go:64] FLAG: --runonce="false"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625085    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625088    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625091    2569 flags.go:64] FLAG: --seccomp-default="false"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625094    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625097    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625100    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 28 03:16:48.629462 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625110    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625114    2569 flags.go:64] FLAG: --storage-driver-password="root"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625117    2569 flags.go:64] FLAG: --storage-driver-secure="false"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625120    2569 flags.go:64] FLAG: --storage-driver-table="stats"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625123    2569 flags.go:64] FLAG: --storage-driver-user="root"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625126    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625129    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625132    2569 flags.go:64] FLAG: --system-cgroups=""
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625135    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625140    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625143    2569 flags.go:64] FLAG: --tls-cert-file=""
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625146    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625151    2569 flags.go:64] FLAG: --tls-min-version=""
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625154    2569 flags.go:64] FLAG: --tls-private-key-file=""
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625157    2569 flags.go:64] FLAG: --topology-manager-policy="none"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625160    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625163    2569 flags.go:64] FLAG: --topology-manager-scope="container"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625166    2569 flags.go:64] FLAG: --v="2"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625170    2569 flags.go:64] FLAG: --version="false"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625174    2569 flags.go:64] FLAG: --vmodule=""
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625179    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.625182    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625281    2569 feature_gate.go:328] unrecognized feature gate: Example
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625285    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625289    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 03:16:48.630086 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625292    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625296    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625299    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625302    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625305    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625308    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625311    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625314    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625317    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625319    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625322    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625325    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625327    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625330    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625332    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625335    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625349    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625352    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625355    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625358    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 03:16:48.630686 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625361    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625364    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625366    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625369    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625373    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625377    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625380    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625383    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625386    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625389    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625392    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625396    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625399    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625405    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625408    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625411    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625414    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625416    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625419    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 03:16:48.631188 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625422    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625424    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625427    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625430    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625432    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625435    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625437    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625440    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625443    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625445    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625448    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625450    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625453    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625456    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625459    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625461    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625464    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625468    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625471    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 03:16:48.631684 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625474    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625478    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625480    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625483    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625485    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625488    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625491    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625495    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625498    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625501    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625503    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625506    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625508    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625511    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625514    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625517    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625520    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625522    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625525    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625527    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 03:16:48.632184 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625530    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 03:16:48.632700 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625532    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 03:16:48.632700 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625535    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 03:16:48.632700 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625537    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 03:16:48.632700 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.625540    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 03:16:48.632700 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.626387    2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 03:16:48.632921 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.632899    2569 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 28 03:16:48.632952 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.632922    2569 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 28 03:16:48.632982 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.632978    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.632983    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.632986    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.632989    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.632992    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.632995    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.632997    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633000    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633003    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633006    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633009    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 03:16:48.633012 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633013    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633018    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633021    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633024    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633027    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633030    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633032    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633035    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633038    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633040    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633043    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633045    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633048    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633050    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633053    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633055    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633058    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633062    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633066    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633069    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 03:16:48.633294 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633077    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633080    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633083    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633086    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633088    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633091    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633094    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633096    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633099    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633101    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633112    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633115    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633117    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633120    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633123    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633126    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633129    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633132    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633134    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633137    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 03:16:48.633799 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633140    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633143    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633145    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633148    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633150    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633152    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633155    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633158    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633161    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633164    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633166    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633168    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633171    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633180    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633183    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633186    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633188    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633191    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633194    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633197    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 03:16:48.634308 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633199    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633202    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633205    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633207    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633210    2569 feature_gate.go:328] unrecognized feature gate: Example
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633213    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633216    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633219    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633228    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633230    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633233    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633236    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633238    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633241    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633243    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.633248    2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 03:16:48.634806 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633386    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633392    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633395    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633398    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633401    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633404    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633407    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633412    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633416    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633419    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633428    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633431    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633434    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633437    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633440    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633443    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633446    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633449    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633452    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633455    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 03:16:48.635204 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633457    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633461    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633464    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633466    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633469    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633472    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633474    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633477    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633479    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633482    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633485    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633487    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633490    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633492    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633495    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633498    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633500    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633503    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633505    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633508    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 03:16:48.635745 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633511    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633513    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633516    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633524    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633527    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633529    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633532    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633535    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633537    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633540    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633542    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633545    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633547    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633550    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633553    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633555    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633558    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633561    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633563    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633566    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 03:16:48.636226 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633569    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633571    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633573    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633576    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633579    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633581    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633584    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633586    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633589    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633591    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633594    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633596    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633599    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633601    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633604    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633607    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633614    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633617    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633620    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 03:16:48.636728 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633622    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 03:16:48.637190 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633625    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 03:16:48.637190 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633627    2569 feature_gate.go:328] unrecognized feature gate: Example
May 28 03:16:48.637190 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633630    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 03:16:48.637190 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633632    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 03:16:48.637190 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633636    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 03:16:48.637190 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:48.633640    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 03:16:48.637190 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.633645    2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 03:16:48.637190 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.634435    2569 server.go:962] "Client rotation is on, will bootstrap in background"
May 28 03:16:48.637405 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.637247    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 28 03:16:48.638397 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.638385    2569 server.go:1019] "Starting client certificate rotation"
May 28 03:16:48.638505 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.638489    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 03:16:48.638566 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.638557    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 03:16:48.665326 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.665304    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 03:16:48.669164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.669135    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 03:16:48.684835 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.684809    2569 log.go:25] "Validated CRI v1 runtime API"
May 28 03:16:48.690139 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.690119    2569 log.go:25] "Validated CRI v1 image API"
May 28 03:16:48.691298 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.691271    2569 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 28 03:16:48.697126 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.697095    2569 fs.go:135] Filesystem UUIDs: map[1a87dcbb-2035-40f8-9a34-7c2c8207b70f:/dev/nvme0n1p4 45ca0d5c-37ea-4f17-a0ae-cd4dd324d20b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
May 28 03:16:48.697214 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.697127    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 28 03:16:48.701577 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.701467    2569 manager.go:217] Machine: {Timestamp:2026-05-28 03:16:48.700957267 +0000 UTC m=+0.416477957 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110039 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2053578973cfd09a5a66eb48272666 SystemUUID:ec205357-8973-cfd0-9a5a-66eb48272666 BootID:f0fbf653-a9fe-4bc5-a12c-f171848c63dc Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b6:43:50:72:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b6:43:50:72:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:7a:53:59:25:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 28 03:16:48.701577 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.701573    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 28 03:16:48.701683 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.701651    2569 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 28 03:16:48.703531 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.703503    2569 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 28 03:16:48.703667 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.703535    2569 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 28 03:16:48.703713 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.703677    2569 topology_manager.go:138] "Creating topology manager with none policy"
May 28 03:16:48.703713 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.703686    2569 container_manager_linux.go:306] "Creating device plugin manager"
May 28 03:16:48.703713 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.703693    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 03:16:48.703713 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.703711    2569 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 03:16:48.705722 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.705711    2569 state_mem.go:36] "Initialized new in-memory state store"
May 28 03:16:48.705828 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.705819    2569 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 28 03:16:48.711056 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.711046    2569 kubelet.go:491] "Attempting to sync node with API server"
May 28 03:16:48.711101 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.711073    2569 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 28 03:16:48.711806 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.711796    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 28 03:16:48.711843 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.711812    2569 kubelet.go:397] "Adding apiserver pod source"
May 28 03:16:48.711843 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.711822    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 28 03:16:48.713042 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.713027    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 03:16:48.713091 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.713054    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 03:16:48.715995 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.715978    2569 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 28 03:16:48.718177 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.718164    2569 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 28 03:16:48.719381 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719368    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 28 03:16:48.719424 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719388    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 28 03:16:48.719424 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719395    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 28 03:16:48.719424 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719400    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 28 03:16:48.719424 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719409    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 28 03:16:48.719424 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719415    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 28 03:16:48.719424 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719420    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 28 03:16:48.719424 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719425    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 28 03:16:48.719608 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719432    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 28 03:16:48.719608 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719438    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 28 03:16:48.719608 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719447    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 28 03:16:48.719608 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.719454    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 28 03:16:48.721130 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.721120    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 28 03:16:48.721130 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.721130    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 28 03:16:48.724577 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.724548    2569 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 28 03:16:48.724677 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.724614    2569 server.go:1295] "Started kubelet"
May 28 03:16:48.725905 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.725190    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 28 03:16:48.725905 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.725808    2569 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 28 03:16:48.725788 ip-10-0-141-29 systemd[1]: Started Kubernetes Kubelet.
May 28 03:16:48.726468 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.725929    2569 server_v1.go:47] "podresources" method="list" useActivePods=true
May 28 03:16:48.727664 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.727643    2569 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 28 03:16:48.727986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.727971    2569 server.go:317] "Adding debug handlers to kubelet server"
May 28 03:16:48.728277 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.728223    2569 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 28 03:16:48.728377 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.728335    2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 28 03:16:48.728433 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.728373    2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 28 03:16:48.733328 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.733306    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 28 03:16:48.733845 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.733814    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 28 03:16:48.734050 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734034    2569 volume_manager.go:295] "The desired_state_of_world populator starts"
May 28 03:16:48.734050 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734051    2569 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 28 03:16:48.734184 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734174    2569 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 28 03:16:48.734233 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734219    2569 reconstruct.go:97] "Volume reconstruction finished"
May 28 03:16:48.734233 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734226    2569 reconciler.go:26] "Reconciler: start to sync state"
May 28 03:16:48.734824 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.733843    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.ec2.internal.18b39c5ecd2c593e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.ec2.internal,UID:ip-10-0-141-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.ec2.internal,},FirstTimestamp:2026-05-28 03:16:48.724572478 +0000 UTC m=+0.440093167,LastTimestamp:2026-05-28 03:16:48.724572478 +0000 UTC m=+0.440093167,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.ec2.internal,}"
May 28 03:16:48.734928 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734895    2569 factory.go:153] Registering CRI-O factory
May 28 03:16:48.734928 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734912    2569 factory.go:223] Registration of the crio container factory successfully
May 28 03:16:48.735016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734968    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 28 03:16:48.735016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734978    2569 factory.go:55] Registering systemd factory
May 28 03:16:48.735016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.734986    2569 factory.go:223] Registration of the systemd container factory successfully
May 28 03:16:48.735016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.735002    2569 factory.go:103] Registering Raw factory
May 28 03:16:48.735016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.735017    2569 manager.go:1196] Started watching for new ooms in manager
May 28 03:16:48.735209 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.735177    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:48.735673 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.735650    2569 manager.go:319] Starting recovery of all containers
May 28 03:16:48.735771 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.735747    2569 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 28 03:16:48.736817 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.736794    2569 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 28 03:16:48.745715 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.745686    2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 03:16:48.746768 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.746752    2569 manager.go:324] Recovery completed
May 28 03:16:48.748232 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.748213    2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
May 28 03:16:48.750517 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.750494    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 28 03:16:48.751274 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.751262    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 03:16:48.753853 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.753837    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
May 28 03:16:48.753915 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.753865    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
May 28 03:16:48.753915 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.753875    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
May 28 03:16:48.754324 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.754310    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 28 03:16:48.754324 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.754322    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 28 03:16:48.754439 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.754337    2569 state_mem.go:36] "Initialized new in-memory state store"
May 28 03:16:48.756773 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.756757    2569 policy_none.go:49] "None policy: Start"
May 28 03:16:48.756773 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.756772    2569 memory_manager.go:186] "Starting memorymanager" policy="None"
May 28 03:16:48.756910 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.756782    2569 state_mem.go:35] "Initializing new in-memory state store"
May 28 03:16:48.769567 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.769495    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.ec2.internal.18b39c5eceeb2159  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.ec2.internal,UID:ip-10-0-141-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-29.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.ec2.internal,},FirstTimestamp:2026-05-28 03:16:48.753852761 +0000 UTC m=+0.469373443,LastTimestamp:2026-05-28 03:16:48.753852761 +0000 UTC m=+0.469373443,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.ec2.internal,}"
May 28 03:16:48.780317 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.780242    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.ec2.internal.18b39c5eceeb63eb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.ec2.internal,UID:ip-10-0-141-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-29.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.ec2.internal,},FirstTimestamp:2026-05-28 03:16:48.753869803 +0000 UTC m=+0.469390486,LastTimestamp:2026-05-28 03:16:48.753869803 +0000 UTC m=+0.469390486,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.ec2.internal,}"
May 28 03:16:48.793894 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.793827    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.ec2.internal.18b39c5eceeb8a8a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.ec2.internal,UID:ip-10-0-141-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-29.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.ec2.internal,},FirstTimestamp:2026-05-28 03:16:48.75387969 +0000 UTC m=+0.469400372,LastTimestamp:2026-05-28 03:16:48.75387969 +0000 UTC m=+0.469400372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.ec2.internal,}"
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.802940    2569 manager.go:341] "Starting Device Plugin manager"
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.803009    2569 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.803023    2569 server.go:85] "Starting device plugin registration server"
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.803269    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.803280    2569 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.803391    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.803471    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.803480    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.804190    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 28 03:16:48.808243 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.804220    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:48.816416 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.816330    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.ec2.internal.18b39c5ed1fcb348  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.ec2.internal,UID:ip-10-0-141-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.ec2.internal,},FirstTimestamp:2026-05-28 03:16:48.80533588 +0000 UTC m=+0.520856550,LastTimestamp:2026-05-28 03:16:48.80533588 +0000 UTC m=+0.520856550,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.ec2.internal,}"
May 28 03:16:48.832421 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.832397    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4h44k"
May 28 03:16:48.847692 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.847669    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4h44k"
May 28 03:16:48.859108 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.859071    2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 28 03:16:48.860285 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.860272    2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 28 03:16:48.860779 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.860765    2569 status_manager.go:230] "Starting to sync pod status with apiserver"
May 28 03:16:48.860850 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.860796    2569 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 28 03:16:48.860850 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.860811    2569 kubelet.go:2451] "Starting kubelet main sync loop"
May 28 03:16:48.860933 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.860919    2569 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 28 03:16:48.901808 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.901730    2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 03:16:48.903680 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.903668    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 03:16:48.904616 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.904594    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
May 28 03:16:48.904707 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.904628    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
May 28 03:16:48.904707 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.904638    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
May 28 03:16:48.904707 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.904661    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-29.ec2.internal"
May 28 03:16:48.914174 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.914150    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-29.ec2.internal"
May 28 03:16:48.914281 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.914180    2569 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-29.ec2.internal\": node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:48.961012 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.960985    2569 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"]
May 28 03:16:48.961153 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.961059    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 03:16:48.962166 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.962149    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
May 28 03:16:48.962255 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.962178    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
May 28 03:16:48.962255 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.962189    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
May 28 03:16:48.963510 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.963498    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 03:16:48.963669 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.963653    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
May 28 03:16:48.963737 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.963701    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 03:16:48.964404 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.964389    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
May 28 03:16:48.964404 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.964398    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
May 28 03:16:48.964498 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.964416    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
May 28 03:16:48.964498 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.964418    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
May 28 03:16:48.964498 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.964428    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
May 28 03:16:48.964498 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.964430    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
May 28 03:16:48.965596 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.965578    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
May 28 03:16:48.965692 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.965606    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 03:16:48.966240 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.966224    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
May 28 03:16:48.966296 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.966254    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
May 28 03:16:48.966296 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:48.966267    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
May 28 03:16:48.977301 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.977280    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:48.995120 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.995093    2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-29.ec2.internal\" not found" node="ip-10-0-141-29.ec2.internal"
May 28 03:16:48.999443 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:48.999428    2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-29.ec2.internal\" not found" node="ip-10-0-141-29.ec2.internal"
May 28 03:16:49.035181 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.035155    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fa96d698c98709a8e74d079ba7e0a5e7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.ec2.internal\" (UID: \"fa96d698c98709a8e74d079ba7e0a5e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.035306 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.035188    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53a896b664f254d5ecdebeda82dce364-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"53a896b664f254d5ecdebeda82dce364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.035306 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.035211    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53a896b664f254d5ecdebeda82dce364-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"53a896b664f254d5ecdebeda82dce364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.078400 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:49.078379    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:49.136131 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.136110    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53a896b664f254d5ecdebeda82dce364-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"53a896b664f254d5ecdebeda82dce364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.136226 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.136139    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53a896b664f254d5ecdebeda82dce364-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"53a896b664f254d5ecdebeda82dce364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.136226 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.136156    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fa96d698c98709a8e74d079ba7e0a5e7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.ec2.internal\" (UID: \"fa96d698c98709a8e74d079ba7e0a5e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.136226 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.136206    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fa96d698c98709a8e74d079ba7e0a5e7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.ec2.internal\" (UID: \"fa96d698c98709a8e74d079ba7e0a5e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.136317 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.136218    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53a896b664f254d5ecdebeda82dce364-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"53a896b664f254d5ecdebeda82dce364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.136317 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.136223    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53a896b664f254d5ecdebeda82dce364-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"53a896b664f254d5ecdebeda82dce364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.179521 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:49.179463    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:49.279780 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:49.279730    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:49.297912 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.297882    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.302698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.302680    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.380805 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:49.380773    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:49.481407 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:49.481336    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:49.582204 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:49.582167    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:49.638553 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.638521    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 28 03:16:49.639195 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.638662    2569 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 03:16:49.682828 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:49.682800    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:49.698913 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.698884    2569 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 03:16:49.733923 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.733681    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 28 03:16:49.761578 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.761547    2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 03:16:49.782641 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:49.782610    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa96d698c98709a8e74d079ba7e0a5e7.slice/crio-1e1edb13e3197f1e18d2ae70998b655977d1672e62316b52034c66df76bd953a WatchSource:0}: Error finding container 1e1edb13e3197f1e18d2ae70998b655977d1672e62316b52034c66df76bd953a: Status 404 returned error can't find the container with id 1e1edb13e3197f1e18d2ae70998b655977d1672e62316b52034c66df76bd953a
May 28 03:16:49.783699 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:49.783675    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:49.788318 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.788304    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 03:16:49.799280 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:49.799255    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a896b664f254d5ecdebeda82dce364.slice/crio-eab9152f68cc285bdbcdbc9cbfaae07dfc4cecf7a1e2dff534ec11343ed422df WatchSource:0}: Error finding container eab9152f68cc285bdbcdbc9cbfaae07dfc4cecf7a1e2dff534ec11343ed422df: Status 404 returned error can't find the container with id eab9152f68cc285bdbcdbc9cbfaae07dfc4cecf7a1e2dff534ec11343ed422df
May 28 03:16:49.845402 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.845370    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v9lrn"
May 28 03:16:49.849400 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.849361    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-27 03:11:48 +0000 UTC" deadline="2027-10-29 19:14:09.940847063 +0000 UTC"
May 28 03:16:49.849465 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.849401    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12471h57m20.091449729s"
May 28 03:16:49.858833 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.858814    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v9lrn"
May 28 03:16:49.864005 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.863966    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal" event={"ID":"53a896b664f254d5ecdebeda82dce364","Type":"ContainerStarted","Data":"eab9152f68cc285bdbcdbc9cbfaae07dfc4cecf7a1e2dff534ec11343ed422df"}
May 28 03:16:49.864988 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.864968    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal" event={"ID":"fa96d698c98709a8e74d079ba7e0a5e7","Type":"ContainerStarted","Data":"1e1edb13e3197f1e18d2ae70998b655977d1672e62316b52034c66df76bd953a"}
May 28 03:16:49.884148 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:49.884120    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
May 28 03:16:49.924800 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.924773    2569 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 03:16:49.929064 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.929043    2569 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 03:16:49.934698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.934673    2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
May 28 03:16:49.964851 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.964829    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 03:16:49.964976 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:49.964941    2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
May 28 03:16:50.001174 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.001099    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 03:16:50.713026 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.712996    2569 apiserver.go:52] "Watching apiserver"
May 28 03:16:50.720203 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.720174    2569 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 28 03:16:50.720610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.720583    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-n7qlq","kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal","openshift-cluster-node-tuning-operator/tuned-hf7th","openshift-multus/multus-additional-cni-plugins-mj6lt","openshift-multus/network-metrics-daemon-4z6ww","openshift-ovn-kubernetes/ovnkube-node-5qfb9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk","openshift-dns/node-resolver-54fkb","openshift-image-registry/node-ca-jjnl2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal","openshift-multus/multus-zhtxl","openshift-network-diagnostics/network-check-target-5j4lb","openshift-network-operator/iptables-alerter-kz2sk"]
May 28 03:16:50.722761 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.722737    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:16:50.723981 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.723955    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.725288 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.725269    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.726545 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.726523    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.727696 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.727674    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:50.727791 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:50.727744    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:16:50.729072 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.729045    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.729399 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.729380    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 28 03:16:50.729650 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.729633    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 28 03:16:50.729745 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.729696    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 28 03:16:50.729745 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.729735    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-px2hd\""
May 28 03:16:50.730556 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.730536    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.732132 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.732114    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.733583 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.733565    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.735192 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.735165    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.736273 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.736250    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 28 03:16:50.736435 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.736387    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 28 03:16:50.737475 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.736885    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 28 03:16:50.737475 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.737043    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 28 03:16:50.737475 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.737458    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 28 03:16:50.737663 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.737457    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 28 03:16:50.738603 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.737940    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 28 03:16:50.738603 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.738081    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:50.738603 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.738147    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 28 03:16:50.738603 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:50.738165    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:16:50.738603 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.738418    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 28 03:16:50.738603 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.738450    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 28 03:16:50.738922 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.738774    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 28 03:16:50.738970 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.738960    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5cj67\""
May 28 03:16:50.741604 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.741561    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 28 03:16:50.741694 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.741638    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 28 03:16:50.741694 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.741660    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 28 03:16:50.741805 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.741787    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 28 03:16:50.741904 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.741888    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 28 03:16:50.741995 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.741980    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 28 03:16:50.742059 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.742001    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xw529\""
May 28 03:16:50.742059 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.742053    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 28 03:16:50.742296 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.742279    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-95ml4\""
May 28 03:16:50.742386 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.742279    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w86zq\""
May 28 03:16:50.742597 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.742583    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 28 03:16:50.746484 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746466    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.746562 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.746562 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.746562 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-run-netns\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.746709 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-var-lib-kubelet\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.746709 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-tmp\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.746814 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746706    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb65h\" (UniqueName: \"kubernetes.io/projected/64beddb4-335a-4ddb-bc37-07555ed4a8f7-kube-api-access-rb65h\") pod \"node-ca-jjnl2\" (UID: \"64beddb4-335a-4ddb-bc37-07555ed4a8f7\") " pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.746814 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746746    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-etc-kubernetes\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.746814 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746771    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-slash\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.746960 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-tuned\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.746960 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-run-k8s-cni-cncf-io\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.746960 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746862    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-var-lib-kubelet\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.746960 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746903    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-node-log\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.746960 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746939    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-lib-modules\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.747170 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.746980    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-device-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.747170 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747009    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566mq\" (UniqueName: \"kubernetes.io/projected/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-kube-api-access-566mq\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.747170 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d-konnectivity-ca\") pod \"konnectivity-agent-n7qlq\" (UID: \"d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d\") " pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:16:50.747170 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfbk\" (UniqueName: \"kubernetes.io/projected/5e793ecb-5a38-43a6-ae37-fa879acec3ea-kube-api-access-kvfbk\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.747170 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747081    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-log-socket\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.747170 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-run-netns\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.747170 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db69a13b-da03-43b3-af25-a070cfe57890-multus-daemon-config\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.747170 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747159    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-os-release\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747175    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747230    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-modprobe-d\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747256    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-systemd\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747286    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64beddb4-335a-4ddb-bc37-07555ed4a8f7-serviceca\") pod \"node-ca-jjnl2\" (UID: \"64beddb4-335a-4ddb-bc37-07555ed4a8f7\") " pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-etc-openvswitch\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-run-ovn\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-ovn-node-metrics-cert\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747417    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k2xb\" (UniqueName: \"kubernetes.io/projected/c9e1ada3-6207-4ebb-84d1-edf6e12f7311-kube-api-access-8k2xb\") pod \"node-resolver-54fkb\" (UID: \"c9e1ada3-6207-4ebb-84d1-edf6e12f7311\") " pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747470    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-run-systemd\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747501    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-run-openvswitch\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747525    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-cni-bin\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.747542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747543    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-sysconfig\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747560    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-kubernetes\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747581    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-sysctl-conf\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-sys-fs\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db69a13b-da03-43b3-af25-a070cfe57890-cni-binary-copy\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747668    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-systemd-units\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747705    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-env-overrides\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747731    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64beddb4-335a-4ddb-bc37-07555ed4a8f7-host\") pod \"node-ca-jjnl2\" (UID: \"64beddb4-335a-4ddb-bc37-07555ed4a8f7\") " pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747766    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75374da5-00d1-4ec3-9e4a-66da41acc109-iptables-alerter-script\") pod \"iptables-alerter-kz2sk\" (UID: \"75374da5-00d1-4ec3-9e4a-66da41acc109\") " pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-cnibin\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-multus-socket-dir-parent\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747833    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-ovnkube-script-lib\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-sysctl-d\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747895    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-sys\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9e1ada3-6207-4ebb-84d1-edf6e12f7311-hosts-file\") pod \"node-resolver-54fkb\" (UID: \"c9e1ada3-6207-4ebb-84d1-edf6e12f7311\") " pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747948    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d-agent-certs\") pod \"konnectivity-agent-n7qlq\" (UID: \"d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d\") " pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88xv\" (UniqueName: \"kubernetes.io/projected/75374da5-00d1-4ec3-9e4a-66da41acc109-kube-api-access-f88xv\") pod \"iptables-alerter-kz2sk\" (UID: \"75374da5-00d1-4ec3-9e4a-66da41acc109\") " pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.748204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.747992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-os-release\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-run-multus-certs\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcz5f\" (UniqueName: \"kubernetes.io/projected/db69a13b-da03-43b3-af25-a070cfe57890-kube-api-access-mcz5f\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748058    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-ovnkube-config\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e1ada3-6207-4ebb-84d1-edf6e12f7311-tmp-dir\") pod \"node-resolver-54fkb\" (UID: \"c9e1ada3-6207-4ebb-84d1-edf6e12f7311\") " pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748179    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75374da5-00d1-4ec3-9e4a-66da41acc109-host-slash\") pod \"iptables-alerter-kz2sk\" (UID: \"75374da5-00d1-4ec3-9e4a-66da41acc109\") " pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748210    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-system-cni-dir\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748236    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-kubelet\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-run\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-registration-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-hostroot\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748359    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748405    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdqsh\" (UniqueName: \"kubernetes.io/projected/1025fde6-0949-48f6-af56-3ecd15e95ff7-kube-api-access-gdqsh\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748427    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-multus-cni-dir\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748449    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-var-lib-cni-multus\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.749000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748487    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-multus-conf-dir\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748531    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-cni-netd\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-host\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748591    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-system-cni-dir\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748626    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-var-lib-openvswitch\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9th6\" (UniqueName: \"kubernetes.io/projected/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-kube-api-access-l9th6\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-socket-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748695    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-var-lib-cni-bin\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748728    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-cnibin\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.749695 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.748759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txsb4\" (UniqueName: \"kubernetes.io/projected/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-kube-api-access-txsb4\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.758078 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.757822    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pvh9f\""
May 28 03:16:50.758078 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.757883    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 28 03:16:50.758078 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.757922    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 28 03:16:50.758253 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.758187    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sz447\""
May 28 03:16:50.758296 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.758274    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lnw94\""
May 28 03:16:50.758591 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.758415    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rfj8z\""
May 28 03:16:50.758591 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.758469    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 28 03:16:50.758591 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.758471    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 28 03:16:50.758591 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.758491    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 28 03:16:50.835284 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.835258    2569 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 28 03:16:50.849635 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849602    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-tuned\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.849804 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849649    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-run-k8s-cni-cncf-io\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.849804 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849698    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-var-lib-kubelet\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.849804 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849747    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-node-log\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.849804 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849778    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-lib-modules\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.849804 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849774    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-run-k8s-cni-cncf-io\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.850054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849814    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-device-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.850054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849830    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-var-lib-kubelet\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.850054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849893    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-566mq\" (UniqueName: \"kubernetes.io/projected/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-kube-api-access-566mq\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.850054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849913    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-node-log\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.850054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849913    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-device-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.850054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849963    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d-konnectivity-ca\") pod \"konnectivity-agent-n7qlq\" (UID: \"d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d\") " pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:16:50.850054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.849960    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-lib-modules\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.850054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850010    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfbk\" (UniqueName: \"kubernetes.io/projected/5e793ecb-5a38-43a6-ae37-fa879acec3ea-kube-api-access-kvfbk\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.850054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850006    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850106    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-log-socket\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850134    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-run-netns\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850158    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db69a13b-da03-43b3-af25-a070cfe57890-multus-daemon-config\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850181    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-log-socket\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850206    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-os-release\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850236    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-run-netns\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850242    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850297    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850360    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-modprobe-d\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850369    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-os-release\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850398    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-systemd\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850423    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850458    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64beddb4-335a-4ddb-bc37-07555ed4a8f7-serviceca\") pod \"node-ca-jjnl2\" (UID: \"64beddb4-335a-4ddb-bc37-07555ed4a8f7\") " pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.850476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850465    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850510    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-etc-openvswitch\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850545    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-run-ovn\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850579    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d-konnectivity-ca\") pod \"konnectivity-agent-n7qlq\" (UID: \"d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d\") " pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850575    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-ovn-node-metrics-cert\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850588    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-modprobe-d\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850628    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-etc-openvswitch\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850645    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-run-ovn\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850659    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8k2xb\" (UniqueName: \"kubernetes.io/projected/c9e1ada3-6207-4ebb-84d1-edf6e12f7311-kube-api-access-8k2xb\") pod \"node-resolver-54fkb\" (UID: \"c9e1ada3-6207-4ebb-84d1-edf6e12f7311\") " pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850695    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-run-systemd\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850722    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-run-openvswitch\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850764    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-cni-bin\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850787    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-sysconfig\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850804    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db69a13b-da03-43b3-af25-a070cfe57890-multus-daemon-config\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850832    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-systemd\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850837    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-sysconfig\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850861    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-run-openvswitch\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850879    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-cni-bin\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.851102 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850886    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-kubernetes\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850897    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-run-systemd\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850920    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-sysctl-conf\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850925    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-kubernetes\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850948    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-sys-fs\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850970    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db69a13b-da03-43b3-af25-a070cfe57890-cni-binary-copy\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.850989    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-systemd-units\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851012    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-env-overrides\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851027    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-sysctl-conf\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851035    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-sys-fs\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851037    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64beddb4-335a-4ddb-bc37-07555ed4a8f7-host\") pod \"node-ca-jjnl2\" (UID: \"64beddb4-335a-4ddb-bc37-07555ed4a8f7\") " pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851071    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64beddb4-335a-4ddb-bc37-07555ed4a8f7-host\") pod \"node-ca-jjnl2\" (UID: \"64beddb4-335a-4ddb-bc37-07555ed4a8f7\") " pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851087    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75374da5-00d1-4ec3-9e4a-66da41acc109-iptables-alerter-script\") pod \"iptables-alerter-kz2sk\" (UID: \"75374da5-00d1-4ec3-9e4a-66da41acc109\") " pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851111    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-systemd-units\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851119    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-cnibin\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851144    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-multus-socket-dir-parent\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851170    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-ovnkube-script-lib\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851195    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-sysctl-d\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.852004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851218    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-sys\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851242    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9e1ada3-6207-4ebb-84d1-edf6e12f7311-hosts-file\") pod \"node-resolver-54fkb\" (UID: \"c9e1ada3-6207-4ebb-84d1-edf6e12f7311\") " pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851267    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d-agent-certs\") pod \"konnectivity-agent-n7qlq\" (UID: \"d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d\") " pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851272    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-multus-socket-dir-parent\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851292    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f88xv\" (UniqueName: \"kubernetes.io/projected/75374da5-00d1-4ec3-9e4a-66da41acc109-kube-api-access-f88xv\") pod \"iptables-alerter-kz2sk\" (UID: \"75374da5-00d1-4ec3-9e4a-66da41acc109\") " pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851315    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-os-release\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851365    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-run-multus-certs\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851390    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mcz5f\" (UniqueName: \"kubernetes.io/projected/db69a13b-da03-43b3-af25-a070cfe57890-kube-api-access-mcz5f\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851411    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-ovnkube-config\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851434    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e1ada3-6207-4ebb-84d1-edf6e12f7311-tmp-dir\") pod \"node-resolver-54fkb\" (UID: \"c9e1ada3-6207-4ebb-84d1-edf6e12f7311\") " pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851456    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75374da5-00d1-4ec3-9e4a-66da41acc109-host-slash\") pod \"iptables-alerter-kz2sk\" (UID: \"75374da5-00d1-4ec3-9e4a-66da41acc109\") " pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851478    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-system-cni-dir\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851499    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-kubelet\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851519    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-run\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851539    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-registration-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851563    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-hostroot\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851588    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.852825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851621    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851647    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851681    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gdqsh\" (UniqueName: \"kubernetes.io/projected/1025fde6-0949-48f6-af56-3ecd15e95ff7-kube-api-access-gdqsh\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851707    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-multus-cni-dir\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851734    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-var-lib-cni-multus\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851781    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-multus-conf-dir\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851845    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-cni-netd\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851871    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-host\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851898    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-system-cni-dir\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851927    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-var-lib-openvswitch\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851955    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l9th6\" (UniqueName: \"kubernetes.io/projected/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-kube-api-access-l9th6\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851985    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852009    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-socket-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852032    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852055    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-var-lib-cni-bin\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852077    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-cnibin\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852102    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-txsb4\" (UniqueName: \"kubernetes.io/projected/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-kube-api-access-txsb4\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.853610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852101    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e1ada3-6207-4ebb-84d1-edf6e12f7311-tmp-dir\") pod \"node-resolver-54fkb\" (UID: \"c9e1ada3-6207-4ebb-84d1-edf6e12f7311\") " pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852124    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852147    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852170    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852190    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db69a13b-da03-43b3-af25-a070cfe57890-cni-binary-copy\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852224    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-env-overrides\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852243    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-system-cni-dir\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852246    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75374da5-00d1-4ec3-9e4a-66da41acc109-host-slash\") pod \"iptables-alerter-kz2sk\" (UID: \"75374da5-00d1-4ec3-9e4a-66da41acc109\") " pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852193    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-run-netns\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852280    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-kubelet\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852288    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-var-lib-kubelet\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852334    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-run\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852356    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-var-lib-kubelet\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852392    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-run-netns\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852428    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-sys\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852444    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-registration-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852490    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-hostroot\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.854471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852520    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-sysctl-d\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852579    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-run-multus-certs\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852610    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-ovnkube-script-lib\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852820    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-os-release\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852871    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-cnibin\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.852994    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853068    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853151    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853180    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e793ecb-5a38-43a6-ae37-fa879acec3ea-socket-dir\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853238    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-multus-cni-dir\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853281    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-var-lib-cni-multus\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853311    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-ovnkube-config\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853322    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-multus-conf-dir\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853401    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-host\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853404    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9e1ada3-6207-4ebb-84d1-edf6e12f7311-hosts-file\") pod \"node-resolver-54fkb\" (UID: \"c9e1ada3-6207-4ebb-84d1-edf6e12f7311\") " pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853445    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-cni-netd\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853452    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-var-lib-openvswitch\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:50.853457    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:50.855164 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853486    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-system-cni-dir\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.851215    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64beddb4-335a-4ddb-bc37-07555ed4a8f7-serviceca\") pod \"node-ca-jjnl2\" (UID: \"64beddb4-335a-4ddb-bc37-07555ed4a8f7\") " pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:50.853550    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs podName:1025fde6-0949-48f6-af56-3ecd15e95ff7 nodeName:}" failed. No retries permitted until 2026-05-28 03:16:51.35351528 +0000 UTC m=+3.069035967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs") pod "network-metrics-daemon-4z6ww" (UID: "1025fde6-0949-48f6-af56-3ecd15e95ff7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853577    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-tmp\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853608    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rb65h\" (UniqueName: \"kubernetes.io/projected/64beddb4-335a-4ddb-bc37-07555ed4a8f7-kube-api-access-rb65h\") pod \"node-ca-jjnl2\" (UID: \"64beddb4-335a-4ddb-bc37-07555ed4a8f7\") " pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853637    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-etc-kubernetes\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.853662    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-slash\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.854885    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75374da5-00d1-4ec3-9e4a-66da41acc109-iptables-alerter-script\") pod \"iptables-alerter-kz2sk\" (UID: \"75374da5-00d1-4ec3-9e4a-66da41acc109\") " pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.855007    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.855049    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.855073    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-etc-tuned\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.855120    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-host-slash\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.855168    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-cnibin\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.855227    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-host-var-lib-cni-bin\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.855514    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d-agent-certs\") pod \"konnectivity-agent-n7qlq\" (UID: \"d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d\") " pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:16:50.855782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.855508    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db69a13b-da03-43b3-af25-a070cfe57890-etc-kubernetes\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.856455 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.855877    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.857124 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.856863    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-ovn-node-metrics-cert\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.857792 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.857770    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-tmp\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.859473 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.859448    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 03:11:49 +0000 UTC" deadline="2028-01-01 09:54:23.70088627 +0000 UTC"
May 28 03:16:50.859473 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.859473    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13998h37m32.841416366s"
May 28 03:16:50.873922 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.873893    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566mq\" (UniqueName: \"kubernetes.io/projected/8d2fdca3-5c87-470f-948f-b6b2b2df48ae-kube-api-access-566mq\") pod \"multus-additional-cni-plugins-mj6lt\" (UID: \"8d2fdca3-5c87-470f-948f-b6b2b2df48ae\") " pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:50.874602 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.874581    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcz5f\" (UniqueName: \"kubernetes.io/projected/db69a13b-da03-43b3-af25-a070cfe57890-kube-api-access-mcz5f\") pod \"multus-zhtxl\" (UID: \"db69a13b-da03-43b3-af25-a070cfe57890\") " pod="openshift-multus/multus-zhtxl"
May 28 03:16:50.875628 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.875608    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9th6\" (UniqueName: \"kubernetes.io/projected/3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de-kube-api-access-l9th6\") pod \"ovnkube-node-5qfb9\" (UID: \"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:50.875804 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.875784    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfbk\" (UniqueName: \"kubernetes.io/projected/5e793ecb-5a38-43a6-ae37-fa879acec3ea-kube-api-access-kvfbk\") pod \"aws-ebs-csi-driver-node-6g4mk\" (UID: \"5e793ecb-5a38-43a6-ae37-fa879acec3ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:50.877071 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.877049    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdqsh\" (UniqueName: \"kubernetes.io/projected/1025fde6-0949-48f6-af56-3ecd15e95ff7-kube-api-access-gdqsh\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:50.884254 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.884227    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb65h\" (UniqueName: \"kubernetes.io/projected/64beddb4-335a-4ddb-bc37-07555ed4a8f7-kube-api-access-rb65h\") pod \"node-ca-jjnl2\" (UID: \"64beddb4-335a-4ddb-bc37-07555ed4a8f7\") " pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:50.890693 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.890673    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsb4\" (UniqueName: \"kubernetes.io/projected/cd1d92f8-78c4-48e8-96ac-c0b6a2671263-kube-api-access-txsb4\") pod \"tuned-hf7th\" (UID: \"cd1d92f8-78c4-48e8-96ac-c0b6a2671263\") " pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:50.895059 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.895038    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k2xb\" (UniqueName: \"kubernetes.io/projected/c9e1ada3-6207-4ebb-84d1-edf6e12f7311-kube-api-access-8k2xb\") pod \"node-resolver-54fkb\" (UID: \"c9e1ada3-6207-4ebb-84d1-edf6e12f7311\") " pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:50.897402 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.897380    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88xv\" (UniqueName: \"kubernetes.io/projected/75374da5-00d1-4ec3-9e4a-66da41acc109-kube-api-access-f88xv\") pod \"iptables-alerter-kz2sk\" (UID: \"75374da5-00d1-4ec3-9e4a-66da41acc109\") " pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:50.943087 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.943058    2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 03:16:50.954308 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:50.954264    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:50.998803 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:50.998697    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 03:16:50.998803 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:50.998732    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 03:16:50.998803 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:50.998745    2569 projected.go:194] Error preparing data for projected volume kube-api-access-xs69l for pod openshift-network-diagnostics/network-check-target-5j4lb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:50.999016 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:50.998821    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l podName:6ce0bce6-c6df-4671-bfad-67499147e7f7 nodeName:}" failed. No retries permitted until 2026-05-28 03:16:51.498799638 +0000 UTC m=+3.214320322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xs69l" (UniqueName: "kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l") pod "network-check-target-5j4lb" (UID: "6ce0bce6-c6df-4671-bfad-67499147e7f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:51.035007 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.034977    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:16:51.044794 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.044771    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kz2sk"
May 28 03:16:51.052439 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.052418    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk"
May 28 03:16:51.057203 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.057180    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mj6lt"
May 28 03:16:51.062826 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.062804    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:16:51.069057 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.069034    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hf7th"
May 28 03:16:51.073747 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.073721    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54fkb"
May 28 03:16:51.080163 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.080133    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjnl2"
May 28 03:16:51.085866 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.085838    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zhtxl"
May 28 03:16:51.282058 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:51.282030    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64beddb4_335a_4ddb_bc37_07555ed4a8f7.slice/crio-ac7d64335f1bd7caf7ccee056d8eb4ceb59b6965ca65bef5c3b515917b5b6398 WatchSource:0}: Error finding container ac7d64335f1bd7caf7ccee056d8eb4ceb59b6965ca65bef5c3b515917b5b6398: Status 404 returned error can't find the container with id ac7d64335f1bd7caf7ccee056d8eb4ceb59b6965ca65bef5c3b515917b5b6398
May 28 03:16:51.283235 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:51.283196    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1d92f8_78c4_48e8_96ac_c0b6a2671263.slice/crio-e405bfefd786b389c7966246ec047c3fffaedbcb92eb2def19892272f3ac6ff6 WatchSource:0}: Error finding container e405bfefd786b389c7966246ec047c3fffaedbcb92eb2def19892272f3ac6ff6: Status 404 returned error can't find the container with id e405bfefd786b389c7966246ec047c3fffaedbcb92eb2def19892272f3ac6ff6
May 28 03:16:51.288213 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:51.288183    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75374da5_00d1_4ec3_9e4a_66da41acc109.slice/crio-e4ac2fc7d3ed2ef5200c7b8807d83e2835ec894f4d9c5ea193a1fb0805d0270d WatchSource:0}: Error finding container e4ac2fc7d3ed2ef5200c7b8807d83e2835ec894f4d9c5ea193a1fb0805d0270d: Status 404 returned error can't find the container with id e4ac2fc7d3ed2ef5200c7b8807d83e2835ec894f4d9c5ea193a1fb0805d0270d
May 28 03:16:51.289223 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:51.289202    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2fdca3_5c87_470f_948f_b6b2b2df48ae.slice/crio-573c5cb385482480ce25091c6e9820f08abf7ecce0d1ab23130a2b087419fa70 WatchSource:0}: Error finding container 573c5cb385482480ce25091c6e9820f08abf7ecce0d1ab23130a2b087419fa70: Status 404 returned error can't find the container with id 573c5cb385482480ce25091c6e9820f08abf7ecce0d1ab23130a2b087419fa70
May 28 03:16:51.289693 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:51.289600    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e793ecb_5a38_43a6_ae37_fa879acec3ea.slice/crio-26ff1159d74584be36080002cabcedb946122ecf13afcaa9d8a73607bc295728 WatchSource:0}: Error finding container 26ff1159d74584be36080002cabcedb946122ecf13afcaa9d8a73607bc295728: Status 404 returned error can't find the container with id 26ff1159d74584be36080002cabcedb946122ecf13afcaa9d8a73607bc295728
May 28 03:16:51.290566 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:51.290544    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a5d0c7_f4a0_472d_9ad2_1680f9855e3d.slice/crio-53f230348a5feec5ac9718bb17f7a90ec75f85081d9de1ff5ea9335130a04e78 WatchSource:0}: Error finding container 53f230348a5feec5ac9718bb17f7a90ec75f85081d9de1ff5ea9335130a04e78: Status 404 returned error can't find the container with id 53f230348a5feec5ac9718bb17f7a90ec75f85081d9de1ff5ea9335130a04e78
May 28 03:16:51.291717 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:51.291420    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb69a13b_da03_43b3_af25_a070cfe57890.slice/crio-9f36761884a26dc4d2fc4e4e0447cf6dd82fbaa635c4f10ede4005e7952fdc82 WatchSource:0}: Error finding container 9f36761884a26dc4d2fc4e4e0447cf6dd82fbaa635c4f10ede4005e7952fdc82: Status 404 returned error can't find the container with id 9f36761884a26dc4d2fc4e4e0447cf6dd82fbaa635c4f10ede4005e7952fdc82
May 28 03:16:51.292603 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:51.292579    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e1ada3_6207_4ebb_84d1_edf6e12f7311.slice/crio-86f4c3bfd333495e92b6a9ed796b874fd4659466eec4770ceeb3a94bccc62a71 WatchSource:0}: Error finding container 86f4c3bfd333495e92b6a9ed796b874fd4659466eec4770ceeb3a94bccc62a71: Status 404 returned error can't find the container with id 86f4c3bfd333495e92b6a9ed796b874fd4659466eec4770ceeb3a94bccc62a71
May 28 03:16:51.294542 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:16:51.294399    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1cabe6_c7fe_4a0c_8b4d_24ca3d2313de.slice/crio-dd1ef9f39a2be4d2f9cc09ee0092dd756b6d71934701ddbd65e869911c6d2b1a WatchSource:0}: Error finding container dd1ef9f39a2be4d2f9cc09ee0092dd756b6d71934701ddbd65e869911c6d2b1a: Status 404 returned error can't find the container with id dd1ef9f39a2be4d2f9cc09ee0092dd756b6d71934701ddbd65e869911c6d2b1a
May 28 03:16:51.357521 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.357495    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:51.357784 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:51.357594    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:51.357784 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:51.357641    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs podName:1025fde6-0949-48f6-af56-3ecd15e95ff7 nodeName:}" failed. No retries permitted until 2026-05-28 03:16:52.357628444 +0000 UTC m=+4.073149113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs") pod "network-metrics-daemon-4z6ww" (UID: "1025fde6-0949-48f6-af56-3ecd15e95ff7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:51.559337 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.559309    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:51.559491 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:51.559469    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 03:16:51.559491 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:51.559494    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 03:16:51.559639 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:51.559506    2569 projected.go:194] Error preparing data for projected volume kube-api-access-xs69l for pod openshift-network-diagnostics/network-check-target-5j4lb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:51.559639 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:51.559554    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l podName:6ce0bce6-c6df-4671-bfad-67499147e7f7 nodeName:}" failed. No retries permitted until 2026-05-28 03:16:52.559539785 +0000 UTC m=+4.275060455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs69l" (UniqueName: "kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l") pod "network-check-target-5j4lb" (UID: "6ce0bce6-c6df-4671-bfad-67499147e7f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:51.870562 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.870490    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 03:11:49 +0000 UTC" deadline="2028-02-12 08:01:23.087477236 +0000 UTC"
May 28 03:16:51.870562 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.870532    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15004h44m31.21694854s"
May 28 03:16:51.881205 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.881172    2569 generic.go:358] "Generic (PLEG): container finished" podID="53a896b664f254d5ecdebeda82dce364" containerID="2ecd28f6294469b6c058781a11c3e24a84868fb80be971cab8dfb13443799c86" exitCode=0
May 28 03:16:51.881375 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.881264    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal" event={"ID":"53a896b664f254d5ecdebeda82dce364","Type":"ContainerDied","Data":"2ecd28f6294469b6c058781a11c3e24a84868fb80be971cab8dfb13443799c86"}
May 28 03:16:51.889158 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.889080    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" event={"ID":"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de","Type":"ContainerStarted","Data":"dd1ef9f39a2be4d2f9cc09ee0092dd756b6d71934701ddbd65e869911c6d2b1a"}
May 28 03:16:51.911787 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.911724    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n7qlq" event={"ID":"d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d","Type":"ContainerStarted","Data":"53f230348a5feec5ac9718bb17f7a90ec75f85081d9de1ff5ea9335130a04e78"}
May 28 03:16:51.921243 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.921162    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk" event={"ID":"5e793ecb-5a38-43a6-ae37-fa879acec3ea","Type":"ContainerStarted","Data":"26ff1159d74584be36080002cabcedb946122ecf13afcaa9d8a73607bc295728"}
May 28 03:16:51.934296 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.934262    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjnl2" event={"ID":"64beddb4-335a-4ddb-bc37-07555ed4a8f7","Type":"ContainerStarted","Data":"ac7d64335f1bd7caf7ccee056d8eb4ceb59b6965ca65bef5c3b515917b5b6398"}
May 28 03:16:51.955999 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.955938    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal" event={"ID":"fa96d698c98709a8e74d079ba7e0a5e7","Type":"ContainerStarted","Data":"259a862ca559e313fbe00c6e3d5bef1e34f48480d0e9cbf8a992013d3ce298b9"}
May 28 03:16:51.974288 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.974250    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj6lt" event={"ID":"8d2fdca3-5c87-470f-948f-b6b2b2df48ae","Type":"ContainerStarted","Data":"573c5cb385482480ce25091c6e9820f08abf7ecce0d1ab23130a2b087419fa70"}
May 28 03:16:51.994325 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:51.994243    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54fkb" event={"ID":"c9e1ada3-6207-4ebb-84d1-edf6e12f7311","Type":"ContainerStarted","Data":"86f4c3bfd333495e92b6a9ed796b874fd4659466eec4770ceeb3a94bccc62a71"}
May 28 03:16:52.008564 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.005300    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zhtxl" event={"ID":"db69a13b-da03-43b3-af25-a070cfe57890","Type":"ContainerStarted","Data":"9f36761884a26dc4d2fc4e4e0447cf6dd82fbaa635c4f10ede4005e7952fdc82"}
May 28 03:16:52.014160 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.014125    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kz2sk" event={"ID":"75374da5-00d1-4ec3-9e4a-66da41acc109","Type":"ContainerStarted","Data":"e4ac2fc7d3ed2ef5200c7b8807d83e2835ec894f4d9c5ea193a1fb0805d0270d"}
May 28 03:16:52.024654 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.024615    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hf7th" event={"ID":"cd1d92f8-78c4-48e8-96ac-c0b6a2671263","Type":"ContainerStarted","Data":"e405bfefd786b389c7966246ec047c3fffaedbcb92eb2def19892272f3ac6ff6"}
May 28 03:16:52.377011 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.376710    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:52.377011 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.376893    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:52.377011 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.376951    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs podName:1025fde6-0949-48f6-af56-3ecd15e95ff7 nodeName:}" failed. No retries permitted until 2026-05-28 03:16:54.37693258 +0000 UTC m=+6.092453256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs") pod "network-metrics-daemon-4z6ww" (UID: "1025fde6-0949-48f6-af56-3ecd15e95ff7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:52.382143 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.382084    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal" podStartSLOduration=3.382065701 podStartE2EDuration="3.382065701s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 03:16:52.005938722 +0000 UTC m=+3.721459415" watchObservedRunningTime="2026-05-28 03:16:52.382065701 +0000 UTC m=+4.097586396"
May 28 03:16:52.382376 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.382232    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-s4vgs"]
May 28 03:16:52.389827 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.389791    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:52.389973 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.389880    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:16:52.477582 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.477547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/442d53f6-d855-4957-9f81-42f46bbb30ce-dbus\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:52.477745 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.477627    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/442d53f6-d855-4957-9f81-42f46bbb30ce-kubelet-config\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:52.477745 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.477680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:52.578000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.577964    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/442d53f6-d855-4957-9f81-42f46bbb30ce-dbus\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:52.578184 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.578034    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:52.578184 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.578064    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/442d53f6-d855-4957-9f81-42f46bbb30ce-kubelet-config\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:52.578184 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.578120    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:52.578368 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.578246    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 03:16:52.578368 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.578304    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret podName:442d53f6-d855-4957-9f81-42f46bbb30ce nodeName:}" failed. No retries permitted until 2026-05-28 03:16:53.078285523 +0000 UTC m=+4.793806198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret") pod "global-pull-secret-syncer-s4vgs" (UID: "442d53f6-d855-4957-9f81-42f46bbb30ce") : object "kube-system"/"original-pull-secret" not registered
May 28 03:16:52.578613 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.578586    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/442d53f6-d855-4957-9f81-42f46bbb30ce-dbus\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:52.578725 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.578704    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 03:16:52.578725 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.578722    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 03:16:52.578857 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.578735    2569 projected.go:194] Error preparing data for projected volume kube-api-access-xs69l for pod openshift-network-diagnostics/network-check-target-5j4lb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:52.578857 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.578781    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l podName:6ce0bce6-c6df-4671-bfad-67499147e7f7 nodeName:}" failed. No retries permitted until 2026-05-28 03:16:54.578764347 +0000 UTC m=+6.294285023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs69l" (UniqueName: "kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l") pod "network-check-target-5j4lb" (UID: "6ce0bce6-c6df-4671-bfad-67499147e7f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:52.578857 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.578831    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/442d53f6-d855-4957-9f81-42f46bbb30ce-kubelet-config\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:52.862993 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.862502    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:52.862993 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.862634    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:16:52.863237 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:52.863040    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:52.863237 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:52.863159    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:16:53.036028 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:53.035914    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal" event={"ID":"53a896b664f254d5ecdebeda82dce364","Type":"ContainerStarted","Data":"12a6765923b6f3140ef332f92b80c446ce6779e6a834208cff6d8b4aa033fbc0"}
May 28 03:16:53.082169 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:53.082132    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:53.082378 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:53.082305    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 03:16:53.082443 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:53.082382    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret podName:442d53f6-d855-4957-9f81-42f46bbb30ce nodeName:}" failed. No retries permitted until 2026-05-28 03:16:54.082362503 +0000 UTC m=+5.797883176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret") pod "global-pull-secret-syncer-s4vgs" (UID: "442d53f6-d855-4957-9f81-42f46bbb30ce") : object "kube-system"/"original-pull-secret" not registered
May 28 03:16:53.861496 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:53.861374    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:53.861667 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:53.861507    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:16:54.091510 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:54.091454    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:54.091974 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.091590    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 03:16:54.091974 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.091651    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret podName:442d53f6-d855-4957-9f81-42f46bbb30ce nodeName:}" failed. No retries permitted until 2026-05-28 03:16:56.091632425 +0000 UTC m=+7.807153094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret") pod "global-pull-secret-syncer-s4vgs" (UID: "442d53f6-d855-4957-9f81-42f46bbb30ce") : object "kube-system"/"original-pull-secret" not registered
May 28 03:16:54.395141 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:54.394620    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:54.395141 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.394750    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:54.395141 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.394813    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs podName:1025fde6-0949-48f6-af56-3ecd15e95ff7 nodeName:}" failed. No retries permitted until 2026-05-28 03:16:58.39479395 +0000 UTC m=+10.110314620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs") pod "network-metrics-daemon-4z6ww" (UID: "1025fde6-0949-48f6-af56-3ecd15e95ff7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:54.596816 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:54.596175    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:54.596816 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.596395    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 03:16:54.596816 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.596413    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 03:16:54.596816 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.596428    2569 projected.go:194] Error preparing data for projected volume kube-api-access-xs69l for pod openshift-network-diagnostics/network-check-target-5j4lb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:54.596816 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.596489    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l podName:6ce0bce6-c6df-4671-bfad-67499147e7f7 nodeName:}" failed. No retries permitted until 2026-05-28 03:16:58.596470588 +0000 UTC m=+10.311991263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs69l" (UniqueName: "kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l") pod "network-check-target-5j4lb" (UID: "6ce0bce6-c6df-4671-bfad-67499147e7f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:54.861868 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:54.861834    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:54.862034 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.861955    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:16:54.862034 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:54.861842    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:54.862135 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:54.862102    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:16:55.861604 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:55.861099    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:55.861604 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:55.861248    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:16:56.111155 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:56.111108    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:56.111383 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:56.111275    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 03:16:56.111383 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:56.111368    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret podName:442d53f6-d855-4957-9f81-42f46bbb30ce nodeName:}" failed. No retries permitted until 2026-05-28 03:17:00.111332573 +0000 UTC m=+11.826853247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret") pod "global-pull-secret-syncer-s4vgs" (UID: "442d53f6-d855-4957-9f81-42f46bbb30ce") : object "kube-system"/"original-pull-secret" not registered
May 28 03:16:56.861630 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:56.861463    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:56.861630 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:56.861608    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:16:56.862224 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:56.861639    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:56.862224 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:56.861714    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:16:57.861409 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:57.861369    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:57.861601 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:57.861500    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:16:58.430923 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:58.430878    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:58.431375 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:58.431048    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:58.431375 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:58.431122    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs podName:1025fde6-0949-48f6-af56-3ecd15e95ff7 nodeName:}" failed. No retries permitted until 2026-05-28 03:17:06.431104806 +0000 UTC m=+18.146625476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs") pod "network-metrics-daemon-4z6ww" (UID: "1025fde6-0949-48f6-af56-3ecd15e95ff7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:16:58.632540 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:58.632497    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:58.632745 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:58.632653    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 03:16:58.632745 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:58.632680    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 03:16:58.632745 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:58.632694    2569 projected.go:194] Error preparing data for projected volume kube-api-access-xs69l for pod openshift-network-diagnostics/network-check-target-5j4lb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:58.632888 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:58.632763    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l podName:6ce0bce6-c6df-4671-bfad-67499147e7f7 nodeName:}" failed. No retries permitted until 2026-05-28 03:17:06.632742503 +0000 UTC m=+18.348263188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs69l" (UniqueName: "kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l") pod "network-check-target-5j4lb" (UID: "6ce0bce6-c6df-4671-bfad-67499147e7f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:16:58.863234 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:58.862567    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:16:58.863234 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:58.862710    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:16:58.863234 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:58.863081    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:16:58.863234 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:58.863172    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:16:59.861418 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:16:59.861045    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:16:59.861418 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:16:59.861438    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:00.147250 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:00.147134    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:00.147485 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:00.147295    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 03:17:00.147485 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:00.147374    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret podName:442d53f6-d855-4957-9f81-42f46bbb30ce nodeName:}" failed. No retries permitted until 2026-05-28 03:17:08.147337535 +0000 UTC m=+19.862858213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret") pod "global-pull-secret-syncer-s4vgs" (UID: "442d53f6-d855-4957-9f81-42f46bbb30ce") : object "kube-system"/"original-pull-secret" not registered
May 28 03:17:00.861775 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:00.861741    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:00.862215 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:00.861863    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:00.862215 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:00.861936    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:00.862215 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:00.862051    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:01.862041 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:01.862000    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:01.862507 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:01.862147    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:02.861614 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:02.861579    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:02.861614 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:02.861600    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:02.861873 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:02.861698    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:02.861873 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:02.861818    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:03.861478 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:03.861450    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:03.861907 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:03.861564    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:04.861220 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:04.861181    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:04.861405 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:04.861181    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:04.861405 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:04.861379    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:04.861524 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:04.861431    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:05.861846 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:05.861810    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:05.862309 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:05.861924    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:06.492769 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:06.492719    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:06.492926 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:06.492867    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:17:06.492974 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:06.492947    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs podName:1025fde6-0949-48f6-af56-3ecd15e95ff7 nodeName:}" failed. No retries permitted until 2026-05-28 03:17:22.492926469 +0000 UTC m=+34.208447145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs") pod "network-metrics-daemon-4z6ww" (UID: "1025fde6-0949-48f6-af56-3ecd15e95ff7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:17:06.694143 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:06.694104    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:06.694392 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:06.694297    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 03:17:06.694392 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:06.694325    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 03:17:06.694392 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:06.694336    2569 projected.go:194] Error preparing data for projected volume kube-api-access-xs69l for pod openshift-network-diagnostics/network-check-target-5j4lb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:17:06.694509 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:06.694406    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l podName:6ce0bce6-c6df-4671-bfad-67499147e7f7 nodeName:}" failed. No retries permitted until 2026-05-28 03:17:22.694391652 +0000 UTC m=+34.409912322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs69l" (UniqueName: "kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l") pod "network-check-target-5j4lb" (UID: "6ce0bce6-c6df-4671-bfad-67499147e7f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:17:06.861642 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:06.861604    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:06.861810 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:06.861602    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:06.861810 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:06.861725    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:06.862214 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:06.861850    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:07.862056 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:07.862018    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:07.862609 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:07.862138    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:08.205150 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:08.205048    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:08.205313 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:08.205215    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 03:17:08.205313 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:08.205297    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret podName:442d53f6-d855-4957-9f81-42f46bbb30ce nodeName:}" failed. No retries permitted until 2026-05-28 03:17:24.20527623 +0000 UTC m=+35.920796920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret") pod "global-pull-secret-syncer-s4vgs" (UID: "442d53f6-d855-4957-9f81-42f46bbb30ce") : object "kube-system"/"original-pull-secret" not registered
May 28 03:17:08.864188 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:08.863552    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:08.864618 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:08.864225    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:08.864618 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:08.864599    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:08.864707 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:08.864606    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:09.861235 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:09.861057    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:09.861358 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:09.861310    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:10.069287 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.069246    2569 generic.go:358] "Generic (PLEG): container finished" podID="8d2fdca3-5c87-470f-948f-b6b2b2df48ae" containerID="58554f8b2bc19834e8de5268d7c56d6c0aa81f498ffa7d3a41010282b78e03b1" exitCode=0
May 28 03:17:10.070143 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.069368    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj6lt" event={"ID":"8d2fdca3-5c87-470f-948f-b6b2b2df48ae","Type":"ContainerDied","Data":"58554f8b2bc19834e8de5268d7c56d6c0aa81f498ffa7d3a41010282b78e03b1"}
May 28 03:17:10.071029 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.070762    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54fkb" event={"ID":"c9e1ada3-6207-4ebb-84d1-edf6e12f7311","Type":"ContainerStarted","Data":"4443f0b444567935aca5bb762c341dcf8b5a9808e5f65032e6b4c77d76767597"}
May 28 03:17:10.072421 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.072318    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zhtxl" event={"ID":"db69a13b-da03-43b3-af25-a070cfe57890","Type":"ContainerStarted","Data":"7d0e9574a454144f049bb7c1bfc9d740efe8543e952d9d1a2fe3f1cbc98c6e2b"}
May 28 03:17:10.073966 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.073943    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hf7th" event={"ID":"cd1d92f8-78c4-48e8-96ac-c0b6a2671263","Type":"ContainerStarted","Data":"0c4c1f01dab06c482b73717e0291f2d7da45a22414e428283ac70703c78573a9"}
May 28 03:17:10.076525 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.076497    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" event={"ID":"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de","Type":"ContainerStarted","Data":"7d15545961f167500b9dc17d1ea3cfe241013c05967e58d0ab28d2c9d77be655"}
May 28 03:17:10.076627 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.076533    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" event={"ID":"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de","Type":"ContainerStarted","Data":"ac7ee7a677c3f25cfd0c582c0a72f8fed1b484171f13eb8595231e9cfb061e7e"}
May 28 03:17:10.076627 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.076547    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" event={"ID":"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de","Type":"ContainerStarted","Data":"0c4f572970ec9f1fbe86a2153ad7b7322c9204ab33c6b4156e9cd2cd578d09ea"}
May 28 03:17:10.076627 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.076558    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" event={"ID":"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de","Type":"ContainerStarted","Data":"bed9f6c912144f840a1a2be75ca21e836432ea1d96f4d80541632dd21ef254dd"}
May 28 03:17:10.076627 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.076570    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" event={"ID":"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de","Type":"ContainerStarted","Data":"4c64cb0d70a5362fe3af39d39efa5ce3a301b726689aa616f63ce568a03713ee"}
May 28 03:17:10.077909 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.077887    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n7qlq" event={"ID":"d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d","Type":"ContainerStarted","Data":"971f53078f205a72f86d01437ffafd0f32e8708e3661fa7a3194b17368a556ad"}
May 28 03:17:10.079207 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.079183    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk" event={"ID":"5e793ecb-5a38-43a6-ae37-fa879acec3ea","Type":"ContainerStarted","Data":"c4efd49081d569710c10bd7b6a1a406d38f0584a54cb14800b87fb9137188895"}
May 28 03:17:10.080611 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.080579    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjnl2" event={"ID":"64beddb4-335a-4ddb-bc37-07555ed4a8f7","Type":"ContainerStarted","Data":"d7c84a6531cbb9de70176365bcd0cb19667230012d354465a43c75faa24c47cd"}
May 28 03:17:10.120683 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.120225    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal" podStartSLOduration=21.120206161 podStartE2EDuration="21.120206161s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 03:16:53.071686308 +0000 UTC m=+4.787207000" watchObservedRunningTime="2026-05-28 03:17:10.120206161 +0000 UTC m=+21.835726856"
May 28 03:17:10.147065 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.147004    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hf7th" podStartSLOduration=3.072305693 podStartE2EDuration="21.146986481s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="2026-05-28 03:16:51.286120363 +0000 UTC m=+3.001641034" lastFinishedPulling="2026-05-28 03:17:09.36080115 +0000 UTC m=+21.076321822" observedRunningTime="2026-05-28 03:17:10.146892019 +0000 UTC m=+21.862412724" watchObservedRunningTime="2026-05-28 03:17:10.146986481 +0000 UTC m=+21.862507178"
May 28 03:17:10.220818 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.220720    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-54fkb" podStartSLOduration=3.21521371 podStartE2EDuration="21.220704944s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="2026-05-28 03:16:51.294075073 +0000 UTC m=+3.009595758" lastFinishedPulling="2026-05-28 03:17:09.299566308 +0000 UTC m=+21.015086992" observedRunningTime="2026-05-28 03:17:10.186911803 +0000 UTC m=+21.902432497" watchObservedRunningTime="2026-05-28 03:17:10.220704944 +0000 UTC m=+21.936225638"
May 28 03:17:10.220818 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.220797    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jjnl2" podStartSLOduration=3.205421905 podStartE2EDuration="21.220793596s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="2026-05-28 03:16:51.284196829 +0000 UTC m=+2.999717502" lastFinishedPulling="2026-05-28 03:17:09.299568509 +0000 UTC m=+21.015089193" observedRunningTime="2026-05-28 03:17:10.220243719 +0000 UTC m=+21.935764408" watchObservedRunningTime="2026-05-28 03:17:10.220793596 +0000 UTC m=+21.936314287"
May 28 03:17:10.263022 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.262966    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n7qlq" podStartSLOduration=12.444854852 podStartE2EDuration="22.262948986s" podCreationTimestamp="2026-05-28 03:16:48 +0000 UTC" firstStartedPulling="2026-05-28 03:16:51.292569099 +0000 UTC m=+3.008089785" lastFinishedPulling="2026-05-28 03:17:01.110663234 +0000 UTC m=+12.826183919" observedRunningTime="2026-05-28 03:17:10.262110438 +0000 UTC m=+21.977631143" watchObservedRunningTime="2026-05-28 03:17:10.262948986 +0000 UTC m=+21.978469681"
May 28 03:17:10.751451 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.751299    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 28 03:17:10.817016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.816866    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-28T03:17:10.751314528Z","UUID":"910e2c77-cebc-4c8a-99e9-0c2de7cbfdb8","Handler":null,"Name":"","Endpoint":""}
May 28 03:17:10.821725 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.819757    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 28 03:17:10.821725 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.819795    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 28 03:17:10.861621 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.861547    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:10.861897 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:10.861585    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:10.861897 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:10.861753    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:10.861897 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:10.861863    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:11.084479 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:11.084396    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kz2sk" event={"ID":"75374da5-00d1-4ec3-9e4a-66da41acc109","Type":"ContainerStarted","Data":"c11817f952f5e80f8497193a466356f39a08af34bc671f5e6a3d9471b44d5511"}
May 28 03:17:11.087622 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:11.087585    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" event={"ID":"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de","Type":"ContainerStarted","Data":"4d8779787be4461ee6f8e4375b2c737d3ae6733989fd93a54103bd6b5c96c891"}
May 28 03:17:11.089259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:11.089218    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk" event={"ID":"5e793ecb-5a38-43a6-ae37-fa879acec3ea","Type":"ContainerStarted","Data":"7a720f9288f52164083906e0d5d26906081887d97b66956dc83cb1ee82b8c48f"}
May 28 03:17:11.143197 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:11.143135    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-zhtxl" podStartSLOduration=4.071913023 podStartE2EDuration="22.143117167s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="2026-05-28 03:16:51.293412064 +0000 UTC m=+3.008932735" lastFinishedPulling="2026-05-28 03:17:09.364616195 +0000 UTC m=+21.080136879" observedRunningTime="2026-05-28 03:17:10.302769041 +0000 UTC m=+22.018289733" watchObservedRunningTime="2026-05-28 03:17:11.143117167 +0000 UTC m=+22.858637897"
May 28 03:17:11.143409 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:11.143335    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kz2sk" podStartSLOduration=5.491113286 podStartE2EDuration="23.143329766s" podCreationTimestamp="2026-05-28 03:16:48 +0000 UTC" firstStartedPulling="2026-05-28 03:16:51.290022776 +0000 UTC m=+3.005543451" lastFinishedPulling="2026-05-28 03:17:08.942239246 +0000 UTC m=+20.657759931" observedRunningTime="2026-05-28 03:17:11.142402206 +0000 UTC m=+22.857922897" watchObservedRunningTime="2026-05-28 03:17:11.143329766 +0000 UTC m=+22.858850459"
May 28 03:17:11.861960 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:11.861918    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:11.862139 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:11.862082    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:12.088788 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:12.088761    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:17:12.089422 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:12.089403    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:17:12.092412 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:12.092381    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk" event={"ID":"5e793ecb-5a38-43a6-ae37-fa879acec3ea","Type":"ContainerStarted","Data":"bdd086b806c7c6fd5f104b75e61f1a250e4768ea4ddc0244a3073c095e0245c6"}
May 28 03:17:12.149519 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:12.149461    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g4mk" podStartSLOduration=2.687078009 podStartE2EDuration="23.149444676s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="2026-05-28 03:16:51.292602373 +0000 UTC m=+3.008123058" lastFinishedPulling="2026-05-28 03:17:11.75496905 +0000 UTC m=+23.470489725" observedRunningTime="2026-05-28 03:17:12.14894495 +0000 UTC m=+23.864465643" watchObservedRunningTime="2026-05-28 03:17:12.149444676 +0000 UTC m=+23.864965367"
May 28 03:17:12.861018 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:12.860986    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:12.861207 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:12.860986    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:12.861207 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:12.861109    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:12.861207 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:12.861188    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:13.097485 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:13.097448    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" event={"ID":"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de","Type":"ContainerStarted","Data":"29a3109b4dab0b105f22af0fdfd8d7a433831f83c278284b7cf45208876da4f7"}
May 28 03:17:13.097979 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:13.097524    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 28 03:17:13.861066 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:13.861029    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:13.861240 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:13.861185    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:14.861398 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:14.861193    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:14.861982 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:14.861196    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:14.861982 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:14.861505    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:14.861982 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:14.861545    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:15.104589 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:15.104555    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" event={"ID":"3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de","Type":"ContainerStarted","Data":"5d768c5174412c3b11a983db444da57df8d134b9cd05c4b04a729457557a5210"}
May 28 03:17:15.105041 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:15.104995    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:17:15.122644 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:15.122576    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:17:15.163973 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:15.163923    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9" podStartSLOduration=7.943982411 podStartE2EDuration="26.163908835s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="2026-05-28 03:16:51.295729709 +0000 UTC m=+3.011250381" lastFinishedPulling="2026-05-28 03:17:09.515656112 +0000 UTC m=+21.231176805" observedRunningTime="2026-05-28 03:17:15.163523172 +0000 UTC m=+26.879043866" watchObservedRunningTime="2026-05-28 03:17:15.163908835 +0000 UTC m=+26.879429527"
May 28 03:17:15.861042 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:15.861005    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:15.861222 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:15.861137    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:16.107247 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.107212    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:17:16.107772 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.107273    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:17:16.124135 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.124037    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:17:16.464824 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.464731    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5j4lb"]
May 28 03:17:16.464956 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.464854    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:16.464956 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:16.464927    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:16.471048 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.471008    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s4vgs"]
May 28 03:17:16.471217 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.471111    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:16.471280 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:16.471206    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:16.494829 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.494782    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4z6ww"]
May 28 03:17:16.494986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.494949    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:16.495094 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:16.495068    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:16.915821 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.915787    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:17:16.916003 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.915943    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 28 03:17:16.916480 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:16.916458    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n7qlq"
May 28 03:17:17.862051 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:17.861866    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:17.862429 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:17.862120    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:18.112034 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:18.111997    2569 generic.go:358] "Generic (PLEG): container finished" podID="8d2fdca3-5c87-470f-948f-b6b2b2df48ae" containerID="3d19217ca4e09016d127bb6b03d43f46baa16b59a892a9a409f621f1c48527f4" exitCode=0
May 28 03:17:18.112254 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:18.112082    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj6lt" event={"ID":"8d2fdca3-5c87-470f-948f-b6b2b2df48ae","Type":"ContainerDied","Data":"3d19217ca4e09016d127bb6b03d43f46baa16b59a892a9a409f621f1c48527f4"}
May 28 03:17:18.863320 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:18.863277    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:18.864078 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:18.863433    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:18.864078 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:18.863484    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:18.864078 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:18.863605    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:19.861316 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:19.861278    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:19.861481 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:19.861396    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:20.117639 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:20.117552    2569 generic.go:358] "Generic (PLEG): container finished" podID="8d2fdca3-5c87-470f-948f-b6b2b2df48ae" containerID="0a69624c43ecbb9a99584e7a6acf3f0686df7a9f9ca4492dd88ec8d4ad71bf26" exitCode=0
May 28 03:17:20.117639 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:20.117617    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj6lt" event={"ID":"8d2fdca3-5c87-470f-948f-b6b2b2df48ae","Type":"ContainerDied","Data":"0a69624c43ecbb9a99584e7a6acf3f0686df7a9f9ca4492dd88ec8d4ad71bf26"}
May 28 03:17:20.861995 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:20.861962    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:20.862192 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:20.861969    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:20.862192 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:20.862083    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:17:20.862305 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:20.862192    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4vgs" podUID="442d53f6-d855-4957-9f81-42f46bbb30ce"
May 28 03:17:21.861417 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:21.861382    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:21.861770 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:21.861492    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j4lb" podUID="6ce0bce6-c6df-4671-bfad-67499147e7f7"
May 28 03:17:22.069859 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.069831    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeReady"
May 28 03:17:22.070027 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.069960    2569 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 28 03:17:22.122565 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.122484    2569 generic.go:358] "Generic (PLEG): container finished" podID="8d2fdca3-5c87-470f-948f-b6b2b2df48ae" containerID="b212ca48bd348c2d2a057ea76ba55ed5167542405be8c18dbb1aee987e2a35a1" exitCode=0
May 28 03:17:22.122565 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.122529    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj6lt" event={"ID":"8d2fdca3-5c87-470f-948f-b6b2b2df48ae","Type":"ContainerDied","Data":"b212ca48bd348c2d2a057ea76ba55ed5167542405be8c18dbb1aee987e2a35a1"}
May 28 03:17:22.225955 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.225926    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9mc89"]
May 28 03:17:22.241594 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.241553    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b4qwv"]
May 28 03:17:22.241772 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.241620    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.251330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.251309    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 28 03:17:22.255713 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.255699    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:22.256839 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.256819    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 28 03:17:22.256929 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.256888    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-krlmb\""
May 28 03:17:22.259327 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.259307    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 28 03:17:22.259491 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.259315    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 28 03:17:22.259491 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.259421    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 28 03:17:22.259491 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.259449    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vsmkp\""
May 28 03:17:22.278759 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.278732    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b4qwv"]
May 28 03:17:22.293068 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.293041    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9mc89"]
May 28 03:17:22.422009 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.421925    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.422009 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.421965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89mvc\" (UniqueName: \"kubernetes.io/projected/eb53683e-628a-4184-9360-1ea2c224534f-kube-api-access-89mvc\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.422009 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.422001    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:22.422228 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.422028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb53683e-628a-4184-9360-1ea2c224534f-config-volume\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.422228 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.422045    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb53683e-628a-4184-9360-1ea2c224534f-tmp-dir\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.422228 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.422065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m82vk\" (UniqueName: \"kubernetes.io/projected/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-kube-api-access-m82vk\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:22.522895 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.522858    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.522895 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.522895    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89mvc\" (UniqueName: \"kubernetes.io/projected/eb53683e-628a-4184-9360-1ea2c224534f-kube-api-access-89mvc\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.523112 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.522915    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:22.523112 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.522976    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:22.523112 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.523014    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 03:17:22.523112 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.523009    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb53683e-628a-4184-9360-1ea2c224534f-config-volume\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.523112 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.523063    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb53683e-628a-4184-9360-1ea2c224534f-tmp-dir\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.523112 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.523098    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:17:22.523112 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.523107    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 03:17:22.523447 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.523126    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls podName:eb53683e-628a-4184-9360-1ea2c224534f nodeName:}" failed. No retries permitted until 2026-05-28 03:17:23.023104388 +0000 UTC m=+34.738625060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls") pod "dns-default-9mc89" (UID: "eb53683e-628a-4184-9360-1ea2c224534f") : secret "dns-default-metrics-tls" not found
May 28 03:17:22.523447 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.523099    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m82vk\" (UniqueName: \"kubernetes.io/projected/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-kube-api-access-m82vk\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:22.523447 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.523158    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert podName:8a1688cd-5f8c-4782-9652-9f0ab1e0206d nodeName:}" failed. No retries permitted until 2026-05-28 03:17:23.02314275 +0000 UTC m=+34.738663443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert") pod "ingress-canary-b4qwv" (UID: "8a1688cd-5f8c-4782-9652-9f0ab1e0206d") : secret "canary-serving-cert" not found
May 28 03:17:22.523447 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.523179    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs podName:1025fde6-0949-48f6-af56-3ecd15e95ff7 nodeName:}" failed. No retries permitted until 2026-05-28 03:17:54.52317145 +0000 UTC m=+66.238692128 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs") pod "network-metrics-daemon-4z6ww" (UID: "1025fde6-0949-48f6-af56-3ecd15e95ff7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 03:17:22.523581 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.523547    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb53683e-628a-4184-9360-1ea2c224534f-tmp-dir\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.523705 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.523686    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb53683e-628a-4184-9360-1ea2c224534f-config-volume\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.552477 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.552317    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m82vk\" (UniqueName: \"kubernetes.io/projected/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-kube-api-access-m82vk\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:22.557807 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.557785    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89mvc\" (UniqueName: \"kubernetes.io/projected/eb53683e-628a-4184-9360-1ea2c224534f-kube-api-access-89mvc\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:22.724733 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.724622    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:22.724871 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.724785    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 03:17:22.724871 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.724808    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 03:17:22.724871 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.724818    2569 projected.go:194] Error preparing data for projected volume kube-api-access-xs69l for pod openshift-network-diagnostics/network-check-target-5j4lb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:17:22.724994 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:22.724876    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l podName:6ce0bce6-c6df-4671-bfad-67499147e7f7 nodeName:}" failed. No retries permitted until 2026-05-28 03:17:54.724859982 +0000 UTC m=+66.440380659 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xs69l" (UniqueName: "kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l") pod "network-check-target-5j4lb" (UID: "6ce0bce6-c6df-4671-bfad-67499147e7f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 03:17:22.862064 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.862029    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:22.862526 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.862080    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:22.866612 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.866589    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 28 03:17:22.866746 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.866708    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vzb4w\""
May 28 03:17:22.874215 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:22.874190    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 03:17:23.028001 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:23.027903    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:23.028183 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:23.028036    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 03:17:23.028183 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:23.028060    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:23.028183 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:23.028111    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls podName:eb53683e-628a-4184-9360-1ea2c224534f nodeName:}" failed. No retries permitted until 2026-05-28 03:17:24.028091241 +0000 UTC m=+35.743611911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls") pod "dns-default-9mc89" (UID: "eb53683e-628a-4184-9360-1ea2c224534f") : secret "dns-default-metrics-tls" not found
May 28 03:17:23.028183 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:23.028159    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 03:17:23.028415 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:23.028229    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert podName:8a1688cd-5f8c-4782-9652-9f0ab1e0206d nodeName:}" failed. No retries permitted until 2026-05-28 03:17:24.02821233 +0000 UTC m=+35.743733011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert") pod "ingress-canary-b4qwv" (UID: "8a1688cd-5f8c-4782-9652-9f0ab1e0206d") : secret "canary-serving-cert" not found
May 28 03:17:23.861611 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:23.861577    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:23.866369 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:23.866328    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 03:17:23.866699 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:23.866410    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 03:17:23.866699 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:23.866441    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pw92t\""
May 28 03:17:24.036090 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:24.036042    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:24.036274 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:24.036113    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:24.036274 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:24.036195    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 03:17:24.036274 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:24.036233    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 03:17:24.036274 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:24.036275    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls podName:eb53683e-628a-4184-9360-1ea2c224534f nodeName:}" failed. No retries permitted until 2026-05-28 03:17:26.036254099 +0000 UTC m=+37.751774788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls") pod "dns-default-9mc89" (UID: "eb53683e-628a-4184-9360-1ea2c224534f") : secret "dns-default-metrics-tls" not found
May 28 03:17:24.036477 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:24.036294    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert podName:8a1688cd-5f8c-4782-9652-9f0ab1e0206d nodeName:}" failed. No retries permitted until 2026-05-28 03:17:26.036285541 +0000 UTC m=+37.751806210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert") pod "ingress-canary-b4qwv" (UID: "8a1688cd-5f8c-4782-9652-9f0ab1e0206d") : secret "canary-serving-cert" not found
May 28 03:17:24.238358 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:24.238250    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:24.241018 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:24.240994    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/442d53f6-d855-4957-9f81-42f46bbb30ce-original-pull-secret\") pod \"global-pull-secret-syncer-s4vgs\" (UID: \"442d53f6-d855-4957-9f81-42f46bbb30ce\") " pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:24.380799 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:24.380761    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4vgs"
May 28 03:17:24.612231 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:24.612193    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s4vgs"]
May 28 03:17:24.617094 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:17:24.617066    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442d53f6_d855_4957_9f81_42f46bbb30ce.slice/crio-4053a4e996cb7b2d4aec49506d3147f4ff8f9372138326326a1fef9cd2249991 WatchSource:0}: Error finding container 4053a4e996cb7b2d4aec49506d3147f4ff8f9372138326326a1fef9cd2249991: Status 404 returned error can't find the container with id 4053a4e996cb7b2d4aec49506d3147f4ff8f9372138326326a1fef9cd2249991
May 28 03:17:25.129088 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:25.129055    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s4vgs" event={"ID":"442d53f6-d855-4957-9f81-42f46bbb30ce","Type":"ContainerStarted","Data":"4053a4e996cb7b2d4aec49506d3147f4ff8f9372138326326a1fef9cd2249991"}
May 28 03:17:26.051562 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:26.051517    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:26.051745 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:26.051589    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:26.051745 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:26.051682    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 03:17:26.051745 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:26.051732    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 03:17:26.051873 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:26.051760    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls podName:eb53683e-628a-4184-9360-1ea2c224534f nodeName:}" failed. No retries permitted until 2026-05-28 03:17:30.051741723 +0000 UTC m=+41.767262396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls") pod "dns-default-9mc89" (UID: "eb53683e-628a-4184-9360-1ea2c224534f") : secret "dns-default-metrics-tls" not found
May 28 03:17:26.051873 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:26.051780    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert podName:8a1688cd-5f8c-4782-9652-9f0ab1e0206d nodeName:}" failed. No retries permitted until 2026-05-28 03:17:30.051770676 +0000 UTC m=+41.767291350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert") pod "ingress-canary-b4qwv" (UID: "8a1688cd-5f8c-4782-9652-9f0ab1e0206d") : secret "canary-serving-cert" not found
May 28 03:17:30.076084 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:30.075872    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:30.076501 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:30.076121    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:30.076501 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:30.076016    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 03:17:30.076501 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:30.076224    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls podName:eb53683e-628a-4184-9360-1ea2c224534f nodeName:}" failed. No retries permitted until 2026-05-28 03:17:38.076206665 +0000 UTC m=+49.791727336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls") pod "dns-default-9mc89" (UID: "eb53683e-628a-4184-9360-1ea2c224534f") : secret "dns-default-metrics-tls" not found
May 28 03:17:30.076501 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:30.076230    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 03:17:30.076501 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:30.076284    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert podName:8a1688cd-5f8c-4782-9652-9f0ab1e0206d nodeName:}" failed. No retries permitted until 2026-05-28 03:17:38.076269961 +0000 UTC m=+49.791790630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert") pod "ingress-canary-b4qwv" (UID: "8a1688cd-5f8c-4782-9652-9f0ab1e0206d") : secret "canary-serving-cert" not found
May 28 03:17:32.145897 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:32.145863    2569 generic.go:358] "Generic (PLEG): container finished" podID="8d2fdca3-5c87-470f-948f-b6b2b2df48ae" containerID="bb3986402dca4b6f161b506af255db1313a95b87c6f07e9d2b15bf6af62b69d0" exitCode=0
May 28 03:17:32.146332 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:32.145952    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj6lt" event={"ID":"8d2fdca3-5c87-470f-948f-b6b2b2df48ae","Type":"ContainerDied","Data":"bb3986402dca4b6f161b506af255db1313a95b87c6f07e9d2b15bf6af62b69d0"}
May 28 03:17:32.147359 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:32.147320    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s4vgs" event={"ID":"442d53f6-d855-4957-9f81-42f46bbb30ce","Type":"ContainerStarted","Data":"47d39811648a11b13dd620e6e13040d104cab119b1ae318e2518ec2d0aef6842"}
May 28 03:17:32.273613 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:32.273554    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-s4vgs" podStartSLOduration=33.738675134 podStartE2EDuration="40.273538309s" podCreationTimestamp="2026-05-28 03:16:52 +0000 UTC" firstStartedPulling="2026-05-28 03:17:24.619287341 +0000 UTC m=+36.334808013" lastFinishedPulling="2026-05-28 03:17:31.154150516 +0000 UTC m=+42.869671188" observedRunningTime="2026-05-28 03:17:32.272368909 +0000 UTC m=+43.987889597" watchObservedRunningTime="2026-05-28 03:17:32.273538309 +0000 UTC m=+43.989059001"
May 28 03:17:33.151399 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:33.151366    2569 generic.go:358] "Generic (PLEG): container finished" podID="8d2fdca3-5c87-470f-948f-b6b2b2df48ae" containerID="5eff339eb39153c6ca3536eab1bd40020e3b2d793c718c6834a2d7b5e714a088" exitCode=0
May 28 03:17:33.151835 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:33.151454    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj6lt" event={"ID":"8d2fdca3-5c87-470f-948f-b6b2b2df48ae","Type":"ContainerDied","Data":"5eff339eb39153c6ca3536eab1bd40020e3b2d793c718c6834a2d7b5e714a088"}
May 28 03:17:34.155615 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:34.155578    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj6lt" event={"ID":"8d2fdca3-5c87-470f-948f-b6b2b2df48ae","Type":"ContainerStarted","Data":"00fd25863087bbf5fc60edea16aeceda9b3b3185a03907b834c9be93b6106e06"}
May 28 03:17:38.127712 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:38.127660    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:38.128314 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:38.127726    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:38.128314 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:38.127825    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 03:17:38.128314 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:38.127904    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls podName:eb53683e-628a-4184-9360-1ea2c224534f nodeName:}" failed. No retries permitted until 2026-05-28 03:17:54.127887839 +0000 UTC m=+65.843408508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls") pod "dns-default-9mc89" (UID: "eb53683e-628a-4184-9360-1ea2c224534f") : secret "dns-default-metrics-tls" not found
May 28 03:17:38.128314 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:38.127826    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 03:17:38.128314 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:38.127987    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert podName:8a1688cd-5f8c-4782-9652-9f0ab1e0206d nodeName:}" failed. No retries permitted until 2026-05-28 03:17:54.127967803 +0000 UTC m=+65.843488492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert") pod "ingress-canary-b4qwv" (UID: "8a1688cd-5f8c-4782-9652-9f0ab1e0206d") : secret "canary-serving-cert" not found
May 28 03:17:48.122463 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:48.122432    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qfb9"
May 28 03:17:48.164262 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:48.164204    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mj6lt" podStartSLOduration=19.308688752 podStartE2EDuration="59.164188388s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="2026-05-28 03:16:51.290735183 +0000 UTC m=+3.006255866" lastFinishedPulling="2026-05-28 03:17:31.146234832 +0000 UTC m=+42.861755502" observedRunningTime="2026-05-28 03:17:34.19207 +0000 UTC m=+45.907590695" watchObservedRunningTime="2026-05-28 03:17:48.164188388 +0000 UTC m=+59.879709141"
May 28 03:17:54.148881 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:54.148824    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:17:54.148881 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:54.148892    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:17:54.149306 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:54.148988    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 03:17:54.149306 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:54.148996    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 03:17:54.149306 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:54.149056    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert podName:8a1688cd-5f8c-4782-9652-9f0ab1e0206d nodeName:}" failed. No retries permitted until 2026-05-28 03:18:26.149040929 +0000 UTC m=+97.864561603 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert") pod "ingress-canary-b4qwv" (UID: "8a1688cd-5f8c-4782-9652-9f0ab1e0206d") : secret "canary-serving-cert" not found
May 28 03:17:54.149306 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:54.149071    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls podName:eb53683e-628a-4184-9360-1ea2c224534f nodeName:}" failed. No retries permitted until 2026-05-28 03:18:26.14906513 +0000 UTC m=+97.864585800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls") pod "dns-default-9mc89" (UID: "eb53683e-628a-4184-9360-1ea2c224534f") : secret "dns-default-metrics-tls" not found
May 28 03:17:54.551241 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:54.551142    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:17:54.553790 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:54.553768    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 03:17:54.562062 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:54.562037    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 03:17:54.562145 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:17:54.562102    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs podName:1025fde6-0949-48f6-af56-3ecd15e95ff7 nodeName:}" failed. No retries permitted until 2026-05-28 03:18:58.56208642 +0000 UTC m=+130.277607095 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs") pod "network-metrics-daemon-4z6ww" (UID: "1025fde6-0949-48f6-af56-3ecd15e95ff7") : secret "metrics-daemon-secret" not found
May 28 03:17:54.752182 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:54.752146    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:54.755600 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:54.755582    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 03:17:54.765584 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:54.765561    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 03:17:54.776613 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:54.776583    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs69l\" (UniqueName: \"kubernetes.io/projected/6ce0bce6-c6df-4671-bfad-67499147e7f7-kube-api-access-xs69l\") pod \"network-check-target-5j4lb\" (UID: \"6ce0bce6-c6df-4671-bfad-67499147e7f7\") " pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:55.073678 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:55.073651    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pw92t\""
May 28 03:17:55.081183 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:55.081163    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:55.207990 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:55.207958    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5j4lb"]
May 28 03:17:55.210987 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:17:55.210961    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce0bce6_c6df_4671_bfad_67499147e7f7.slice/crio-09f50ab93248d737e840e4fa0cf189da4424030662407ba57302f46e31dad54b WatchSource:0}: Error finding container 09f50ab93248d737e840e4fa0cf189da4424030662407ba57302f46e31dad54b: Status 404 returned error can't find the container with id 09f50ab93248d737e840e4fa0cf189da4424030662407ba57302f46e31dad54b
May 28 03:17:56.198624 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:56.198583    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5j4lb" event={"ID":"6ce0bce6-c6df-4671-bfad-67499147e7f7","Type":"ContainerStarted","Data":"09f50ab93248d737e840e4fa0cf189da4424030662407ba57302f46e31dad54b"}
May 28 03:17:59.204988 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:59.204947    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5j4lb" event={"ID":"6ce0bce6-c6df-4671-bfad-67499147e7f7","Type":"ContainerStarted","Data":"fce06e5184ddab7519012081bac4d451f6a0382e3369ed77ba8526a9b61f4421"}
May 28 03:17:59.205421 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:59.205121    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:17:59.225111 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:17:59.225057    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5j4lb" podStartSLOduration=67.149549783 podStartE2EDuration="1m10.225041471s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="2026-05-28 03:17:55.212748977 +0000 UTC m=+66.928269650" lastFinishedPulling="2026-05-28 03:17:58.288240661 +0000 UTC m=+70.003761338" observedRunningTime="2026-05-28 03:17:59.223618684 +0000 UTC m=+70.939139387" watchObservedRunningTime="2026-05-28 03:17:59.225041471 +0000 UTC m=+70.940562185"
May 28 03:18:26.164686 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:18:26.164640    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:18:26.165113 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:18:26.164709    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:18:26.165113 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:18:26.164797    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 03:18:26.165113 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:18:26.164857    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls podName:eb53683e-628a-4184-9360-1ea2c224534f nodeName:}" failed. No retries permitted until 2026-05-28 03:19:30.164844419 +0000 UTC m=+161.880365090 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls") pod "dns-default-9mc89" (UID: "eb53683e-628a-4184-9360-1ea2c224534f") : secret "dns-default-metrics-tls" not found
May 28 03:18:26.165113 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:18:26.164795    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 03:18:26.165113 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:18:26.164940    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert podName:8a1688cd-5f8c-4782-9652-9f0ab1e0206d nodeName:}" failed. No retries permitted until 2026-05-28 03:19:30.164924766 +0000 UTC m=+161.880445436 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert") pod "ingress-canary-b4qwv" (UID: "8a1688cd-5f8c-4782-9652-9f0ab1e0206d") : secret "canary-serving-cert" not found
May 28 03:18:30.208416 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:18:30.208383    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5j4lb"
May 28 03:18:58.586671 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:18:58.586626    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:18:58.587152 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:18:58.586779    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 03:18:58.587152 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:18:58.586843    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs podName:1025fde6-0949-48f6-af56-3ecd15e95ff7 nodeName:}" failed. No retries permitted until 2026-05-28 03:21:00.586826043 +0000 UTC m=+252.302346712 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs") pod "network-metrics-daemon-4z6ww" (UID: "1025fde6-0949-48f6-af56-3ecd15e95ff7") : secret "metrics-daemon-secret" not found
May 28 03:19:10.521167 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.521119    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"]
May 28 03:19:10.522943 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.522927    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:10.529413 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.529386    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 28 03:19:10.530134 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.530111    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f758b8f98-x985d"]
May 28 03:19:10.530193 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.530169    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-k4scd\""
May 28 03:19:10.530229 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.530209    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 28 03:19:10.531768 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.531753    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.535996 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.535981    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 28 03:19:10.536224 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.536205    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 28 03:19:10.536457 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.536232    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 28 03:19:10.536457 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.536318    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 28 03:19:10.548998 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.548972    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4cm2m\""
May 28 03:19:10.552853 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.552823    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 28 03:19:10.559097 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.559064    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"]
May 28 03:19:10.560897 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.560874    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f758b8f98-x985d"]
May 28 03:19:10.615306 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.615273    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-bqzm4"]
May 28 03:19:10.620773 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.620740    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.623258 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.623236    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 28 03:19:10.623258 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.623250    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 28 03:19:10.623258 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.623257    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 28 03:19:10.623579 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.623323    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hx4jm\""
May 28 03:19:10.623579 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.623369    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 28 03:19:10.628551 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.628534    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 28 03:19:10.631029 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.631009    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-bqzm4"]
May 28 03:19:10.669958 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.669922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-kube-api-access-8xj66\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.669958 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.669957    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:10.670191 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.669976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152b3248-ef2a-4e6f-924b-609074c896a2-ca-trust-extracted\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.670191 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.669997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-installation-pull-secrets\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.670191 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.670096    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-bound-sa-token\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.670191 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.670142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-registry-certificates\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.670418 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.670209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-image-registry-private-configuration\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.670418 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.670257    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-trusted-ca\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.670418 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.670313    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5sn\" (UniqueName: \"kubernetes.io/projected/5706b72d-d51d-4163-a78a-ce815f2bd4a1-kube-api-access-7q5sn\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:10.670418 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.670399    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.771630 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771546    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-installation-pull-secrets\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.771630 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771586    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-bound-sa-token\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.771630 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04dfd913-171c-498e-b7ae-d73e936b328f-serving-cert\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.772079 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771654    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-registry-certificates\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.772079 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04dfd913-171c-498e-b7ae-d73e936b328f-trusted-ca\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.772079 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8cjj\" (UniqueName: \"kubernetes.io/projected/04dfd913-171c-498e-b7ae-d73e936b328f-kube-api-access-q8cjj\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.772079 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771848    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-image-registry-private-configuration\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.772079 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771880    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-trusted-ca\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.772079 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04dfd913-171c-498e-b7ae-d73e936b328f-config\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.772079 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.771978    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5sn\" (UniqueName: \"kubernetes.io/projected/5706b72d-d51d-4163-a78a-ce815f2bd4a1-kube-api-access-7q5sn\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:10.772079 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.772060    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.772500 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.772098    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-kube-api-access-8xj66\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.772500 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.772125    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:10.772500 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.772156    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152b3248-ef2a-4e6f-924b-609074c896a2-ca-trust-extracted\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.772500 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:10.772193    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 03:19:10.772500 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:10.772212    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f758b8f98-x985d: secret "image-registry-tls" not found
May 28 03:19:10.772500 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:10.772270    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls podName:152b3248-ef2a-4e6f-924b-609074c896a2 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:11.272252337 +0000 UTC m=+142.987773022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls") pod "image-registry-6f758b8f98-x985d" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2") : secret "image-registry-tls" not found
May 28 03:19:10.772500 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.772383    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-registry-certificates\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.772500 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:10.772481    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 03:19:10.772810 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:10.772538    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls podName:5706b72d-d51d-4163-a78a-ce815f2bd4a1 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:11.272520102 +0000 UTC m=+142.988040771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls") pod "cluster-samples-operator-697587599-n9wrr" (UID: "5706b72d-d51d-4163-a78a-ce815f2bd4a1") : secret "samples-operator-tls" not found
May 28 03:19:10.772810 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.772537    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152b3248-ef2a-4e6f-924b-609074c896a2-ca-trust-extracted\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.772810 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.772695    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-trusted-ca\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.774516 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.774493    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-installation-pull-secrets\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.774612 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.774533    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-image-registry-private-configuration\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.783381 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.783331    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-kube-api-access-8xj66\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.783381 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.783376    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-bound-sa-token\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:10.783607 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.783589    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5sn\" (UniqueName: \"kubernetes.io/projected/5706b72d-d51d-4163-a78a-ce815f2bd4a1-kube-api-access-7q5sn\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:10.872651 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.872617    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04dfd913-171c-498e-b7ae-d73e936b328f-serving-cert\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.872839 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.872660    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04dfd913-171c-498e-b7ae-d73e936b328f-trusted-ca\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.872839 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.872693    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8cjj\" (UniqueName: \"kubernetes.io/projected/04dfd913-171c-498e-b7ae-d73e936b328f-kube-api-access-q8cjj\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.872839 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.872722    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04dfd913-171c-498e-b7ae-d73e936b328f-config\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.873431 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.873407    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04dfd913-171c-498e-b7ae-d73e936b328f-config\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.873526 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.873513    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04dfd913-171c-498e-b7ae-d73e936b328f-trusted-ca\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.875000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.874978    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04dfd913-171c-498e-b7ae-d73e936b328f-serving-cert\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.881155 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.881120    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8cjj\" (UniqueName: \"kubernetes.io/projected/04dfd913-171c-498e-b7ae-d73e936b328f-kube-api-access-q8cjj\") pod \"console-operator-76dff48d59-bqzm4\" (UID: \"04dfd913-171c-498e-b7ae-d73e936b328f\") " pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:10.930293 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:10.930262    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:11.049956 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:11.049881    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-bqzm4"]
May 28 03:19:11.052831 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:11.052803    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dfd913_171c_498e_b7ae_d73e936b328f.slice/crio-caad2728b679fe0c69628005e18f8d7c008d6839b36329e19564f04a44806799 WatchSource:0}: Error finding container caad2728b679fe0c69628005e18f8d7c008d6839b36329e19564f04a44806799: Status 404 returned error can't find the container with id caad2728b679fe0c69628005e18f8d7c008d6839b36329e19564f04a44806799
May 28 03:19:11.275931 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:11.275893    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:11.275931 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:11.275937    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:11.276125 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:11.276036    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 03:19:11.276125 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:11.276095    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls podName:5706b72d-d51d-4163-a78a-ce815f2bd4a1 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:12.276080323 +0000 UTC m=+143.991600993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls") pod "cluster-samples-operator-697587599-n9wrr" (UID: "5706b72d-d51d-4163-a78a-ce815f2bd4a1") : secret "samples-operator-tls" not found
May 28 03:19:11.276125 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:11.276036    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 03:19:11.276228 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:11.276131    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f758b8f98-x985d: secret "image-registry-tls" not found
May 28 03:19:11.276228 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:11.276194    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls podName:152b3248-ef2a-4e6f-924b-609074c896a2 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:12.276178699 +0000 UTC m=+143.991699381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls") pod "image-registry-6f758b8f98-x985d" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2") : secret "image-registry-tls" not found
May 28 03:19:11.340099 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:11.340068    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4" event={"ID":"04dfd913-171c-498e-b7ae-d73e936b328f","Type":"ContainerStarted","Data":"caad2728b679fe0c69628005e18f8d7c008d6839b36329e19564f04a44806799"}
May 28 03:19:12.283834 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:12.283794    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:12.284244 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:12.283841    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:12.284244 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:12.283965    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 03:19:12.284244 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:12.283970    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 03:19:12.284244 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:12.283995    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f758b8f98-x985d: secret "image-registry-tls" not found
May 28 03:19:12.284244 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:12.284047    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls podName:5706b72d-d51d-4163-a78a-ce815f2bd4a1 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:14.284027514 +0000 UTC m=+145.999548198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls") pod "cluster-samples-operator-697587599-n9wrr" (UID: "5706b72d-d51d-4163-a78a-ce815f2bd4a1") : secret "samples-operator-tls" not found
May 28 03:19:12.284244 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:12.284066    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls podName:152b3248-ef2a-4e6f-924b-609074c896a2 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:14.284057186 +0000 UTC m=+145.999577857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls") pod "image-registry-6f758b8f98-x985d" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2") : secret "image-registry-tls" not found
May 28 03:19:14.299072 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:14.299024    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:14.299072 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:14.299069    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:14.299530 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:14.299175    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 03:19:14.299530 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:14.299180    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 03:19:14.299530 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:14.299201    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f758b8f98-x985d: secret "image-registry-tls" not found
May 28 03:19:14.299530 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:14.299229    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls podName:5706b72d-d51d-4163-a78a-ce815f2bd4a1 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:18.299213957 +0000 UTC m=+150.014734627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls") pod "cluster-samples-operator-697587599-n9wrr" (UID: "5706b72d-d51d-4163-a78a-ce815f2bd4a1") : secret "samples-operator-tls" not found
May 28 03:19:14.299530 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:14.299242    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls podName:152b3248-ef2a-4e6f-924b-609074c896a2 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:18.299236151 +0000 UTC m=+150.014756820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls") pod "image-registry-6f758b8f98-x985d" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2") : secret "image-registry-tls" not found
May 28 03:19:14.346465 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:14.346440    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/0.log"
May 28 03:19:14.346593 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:14.346478    2569 generic.go:358] "Generic (PLEG): container finished" podID="04dfd913-171c-498e-b7ae-d73e936b328f" containerID="9659d837bc1e2abf37fcdea9efdba85a91331eca5ca860ed9a404b94e5c8693f" exitCode=255
May 28 03:19:14.346593 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:14.346515    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4" event={"ID":"04dfd913-171c-498e-b7ae-d73e936b328f","Type":"ContainerDied","Data":"9659d837bc1e2abf37fcdea9efdba85a91331eca5ca860ed9a404b94e5c8693f"}
May 28 03:19:14.346762 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:14.346749    2569 scope.go:117] "RemoveContainer" containerID="9659d837bc1e2abf37fcdea9efdba85a91331eca5ca860ed9a404b94e5c8693f"
May 28 03:19:15.349303 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:15.349278    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:19:15.349701 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:15.349647    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/0.log"
May 28 03:19:15.349701 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:15.349680    2569 generic.go:358] "Generic (PLEG): container finished" podID="04dfd913-171c-498e-b7ae-d73e936b328f" containerID="f7c9fe2e92b36c5ff02bbbfa928f325ee355f4221af10f2ff8926289ddf16a49" exitCode=255
May 28 03:19:15.349773 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:15.349716    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4" event={"ID":"04dfd913-171c-498e-b7ae-d73e936b328f","Type":"ContainerDied","Data":"f7c9fe2e92b36c5ff02bbbfa928f325ee355f4221af10f2ff8926289ddf16a49"}
May 28 03:19:15.349773 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:15.349753    2569 scope.go:117] "RemoveContainer" containerID="9659d837bc1e2abf37fcdea9efdba85a91331eca5ca860ed9a404b94e5c8693f"
May 28 03:19:15.349997 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:15.349966    2569 scope.go:117] "RemoveContainer" containerID="f7c9fe2e92b36c5ff02bbbfa928f325ee355f4221af10f2ff8926289ddf16a49"
May 28 03:19:15.350165 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:15.350142    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-bqzm4_openshift-console-operator(04dfd913-171c-498e-b7ae-d73e936b328f)\"" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4" podUID="04dfd913-171c-498e-b7ae-d73e936b328f"
May 28 03:19:16.352391 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:16.352364    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:19:16.352773 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:16.352678    2569 scope.go:117] "RemoveContainer" containerID="f7c9fe2e92b36c5ff02bbbfa928f325ee355f4221af10f2ff8926289ddf16a49"
May 28 03:19:16.352847 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:16.352829    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-bqzm4_openshift-console-operator(04dfd913-171c-498e-b7ae-d73e936b328f)\"" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4" podUID="04dfd913-171c-498e-b7ae-d73e936b328f"
May 28 03:19:16.653530 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:16.653458    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54fkb_c9e1ada3-6207-4ebb-84d1-edf6e12f7311/dns-node-resolver/0.log"
May 28 03:19:17.846832 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:17.846806    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jjnl2_64beddb4-335a-4ddb-bc37-07555ed4a8f7/node-ca/0.log"
May 28 03:19:18.330003 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.329912    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:18.330003 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.329957    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:18.330182 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:18.330058    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 03:19:18.330182 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:18.330075    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 03:19:18.330182 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:18.330099    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f758b8f98-x985d: secret "image-registry-tls" not found
May 28 03:19:18.330182 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:18.330110    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls podName:5706b72d-d51d-4163-a78a-ce815f2bd4a1 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:26.330096528 +0000 UTC m=+158.045617197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls") pod "cluster-samples-operator-697587599-n9wrr" (UID: "5706b72d-d51d-4163-a78a-ce815f2bd4a1") : secret "samples-operator-tls" not found
May 28 03:19:18.330182 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:18.330141    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls podName:152b3248-ef2a-4e6f-924b-609074c896a2 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:26.33012857 +0000 UTC m=+158.045649239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls") pod "image-registry-6f758b8f98-x985d" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2") : secret "image-registry-tls" not found
May 28 03:19:18.388776 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.388739    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-z8kpl"]
May 28 03:19:18.391024 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.391008    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.394275 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.394251    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 28 03:19:18.394409 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.394307    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 28 03:19:18.395449 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.395429    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 28 03:19:18.395540 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.395477    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9xwhw\""
May 28 03:19:18.395540 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.395432    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 28 03:19:18.405192 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.405164    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-z8kpl"]
May 28 03:19:18.430812 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.430784    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99a3c63e-4663-4d5a-a2e6-3230e300e789-signing-key\") pod \"service-ca-67d667c7d-z8kpl\" (UID: \"99a3c63e-4663-4d5a-a2e6-3230e300e789\") " pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.430965 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.430848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99a3c63e-4663-4d5a-a2e6-3230e300e789-signing-cabundle\") pod \"service-ca-67d667c7d-z8kpl\" (UID: \"99a3c63e-4663-4d5a-a2e6-3230e300e789\") " pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.430965 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.430880    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv5k6\" (UniqueName: \"kubernetes.io/projected/99a3c63e-4663-4d5a-a2e6-3230e300e789-kube-api-access-bv5k6\") pod \"service-ca-67d667c7d-z8kpl\" (UID: \"99a3c63e-4663-4d5a-a2e6-3230e300e789\") " pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.531615 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.531589    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99a3c63e-4663-4d5a-a2e6-3230e300e789-signing-key\") pod \"service-ca-67d667c7d-z8kpl\" (UID: \"99a3c63e-4663-4d5a-a2e6-3230e300e789\") " pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.531744 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.531652    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99a3c63e-4663-4d5a-a2e6-3230e300e789-signing-cabundle\") pod \"service-ca-67d667c7d-z8kpl\" (UID: \"99a3c63e-4663-4d5a-a2e6-3230e300e789\") " pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.531744 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.531682    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bv5k6\" (UniqueName: \"kubernetes.io/projected/99a3c63e-4663-4d5a-a2e6-3230e300e789-kube-api-access-bv5k6\") pod \"service-ca-67d667c7d-z8kpl\" (UID: \"99a3c63e-4663-4d5a-a2e6-3230e300e789\") " pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.532368 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.532326    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99a3c63e-4663-4d5a-a2e6-3230e300e789-signing-cabundle\") pod \"service-ca-67d667c7d-z8kpl\" (UID: \"99a3c63e-4663-4d5a-a2e6-3230e300e789\") " pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.534007 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.533986    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99a3c63e-4663-4d5a-a2e6-3230e300e789-signing-key\") pod \"service-ca-67d667c7d-z8kpl\" (UID: \"99a3c63e-4663-4d5a-a2e6-3230e300e789\") " pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.545554 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.545534    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv5k6\" (UniqueName: \"kubernetes.io/projected/99a3c63e-4663-4d5a-a2e6-3230e300e789-kube-api-access-bv5k6\") pod \"service-ca-67d667c7d-z8kpl\" (UID: \"99a3c63e-4663-4d5a-a2e6-3230e300e789\") " pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.699669 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.699618    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-z8kpl"
May 28 03:19:18.819986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:18.819950    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-z8kpl"]
May 28 03:19:18.823574 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:18.823540    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a3c63e_4663_4d5a_a2e6_3230e300e789.slice/crio-c647a72a9fcbdd23bc3da8c919aef858115ed9dded12056952fee0632c0b7cc0 WatchSource:0}: Error finding container c647a72a9fcbdd23bc3da8c919aef858115ed9dded12056952fee0632c0b7cc0: Status 404 returned error can't find the container with id c647a72a9fcbdd23bc3da8c919aef858115ed9dded12056952fee0632c0b7cc0
May 28 03:19:19.359098 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:19.359059    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-z8kpl" event={"ID":"99a3c63e-4663-4d5a-a2e6-3230e300e789","Type":"ContainerStarted","Data":"c647a72a9fcbdd23bc3da8c919aef858115ed9dded12056952fee0632c0b7cc0"}
May 28 03:19:20.930891 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:20.930840    2569 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:20.930891 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:20.930895    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:20.931448 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:20.931372    2569 scope.go:117] "RemoveContainer" containerID="f7c9fe2e92b36c5ff02bbbfa928f325ee355f4221af10f2ff8926289ddf16a49"
May 28 03:19:20.931621 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:20.931599    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-bqzm4_openshift-console-operator(04dfd913-171c-498e-b7ae-d73e936b328f)\"" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4" podUID="04dfd913-171c-498e-b7ae-d73e936b328f"
May 28 03:19:21.364543 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:21.364507    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-z8kpl" event={"ID":"99a3c63e-4663-4d5a-a2e6-3230e300e789","Type":"ContainerStarted","Data":"8010bc2745014a94b538dffbf72fadb80e1e6332293a0988b593101e319782cf"}
May 28 03:19:21.393702 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:21.393646    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-z8kpl" podStartSLOduration=1.03383015 podStartE2EDuration="3.393632034s" podCreationTimestamp="2026-05-28 03:19:18 +0000 UTC" firstStartedPulling="2026-05-28 03:19:18.825477087 +0000 UTC m=+150.540997756" lastFinishedPulling="2026-05-28 03:19:21.185278953 +0000 UTC m=+152.900799640" observedRunningTime="2026-05-28 03:19:21.392722097 +0000 UTC m=+153.108242790" watchObservedRunningTime="2026-05-28 03:19:21.393632034 +0000 UTC m=+153.109152725"
May 28 03:19:25.250060 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:25.250010    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9mc89" podUID="eb53683e-628a-4184-9360-1ea2c224534f"
May 28 03:19:25.263205 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:25.263171    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-b4qwv" podUID="8a1688cd-5f8c-4782-9652-9f0ab1e0206d"
May 28 03:19:25.372316 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:25.372280    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:19:25.372523 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:25.372287    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9mc89"
May 28 03:19:25.874585 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:25.874526    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4z6ww" podUID="1025fde6-0949-48f6-af56-3ecd15e95ff7"
May 28 03:19:26.395738 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:26.395696    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:26.395738 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:26.395742    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:26.396188 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:26.395849    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 03:19:26.396188 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:26.395868    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f758b8f98-x985d: secret "image-registry-tls" not found
May 28 03:19:26.396188 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:26.395925    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls podName:152b3248-ef2a-4e6f-924b-609074c896a2 nodeName:}" failed. No retries permitted until 2026-05-28 03:19:42.395909397 +0000 UTC m=+174.111430067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls") pod "image-registry-6f758b8f98-x985d" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2") : secret "image-registry-tls" not found
May 28 03:19:26.398217 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:26.398192    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5706b72d-d51d-4163-a78a-ce815f2bd4a1-samples-operator-tls\") pod \"cluster-samples-operator-697587599-n9wrr\" (UID: \"5706b72d-d51d-4163-a78a-ce815f2bd4a1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:26.431225 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:26.431196    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"
May 28 03:19:26.552595 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:26.552563    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr"]
May 28 03:19:27.378065 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:27.378020    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr" event={"ID":"5706b72d-d51d-4163-a78a-ce815f2bd4a1","Type":"ContainerStarted","Data":"d02490a957094f013b84a52015e79bcb74340b9e4ae684d971588dede664f998"}
May 28 03:19:29.388515 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:29.388474    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr" event={"ID":"5706b72d-d51d-4163-a78a-ce815f2bd4a1","Type":"ContainerStarted","Data":"d9a093c846e080f2227ef755cf47f0bc98ec211b93bfcb5863457d11be52c2d7"}
May 28 03:19:29.388515 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:29.388520    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr" event={"ID":"5706b72d-d51d-4163-a78a-ce815f2bd4a1","Type":"ContainerStarted","Data":"7f21543cfda0ea22c582b6ce1e3c05d5cd27ee48716724efefe548179c60fea3"}
May 28 03:19:29.409209 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:29.409151    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-n9wrr" podStartSLOduration=17.288243763 podStartE2EDuration="19.40913648s" podCreationTimestamp="2026-05-28 03:19:10 +0000 UTC" firstStartedPulling="2026-05-28 03:19:26.594032757 +0000 UTC m=+158.309553430" lastFinishedPulling="2026-05-28 03:19:28.714925474 +0000 UTC m=+160.430446147" observedRunningTime="2026-05-28 03:19:29.408294884 +0000 UTC m=+161.123815576" watchObservedRunningTime="2026-05-28 03:19:29.40913648 +0000 UTC m=+161.124657172"
May 28 03:19:30.223450 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.223411    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:19:30.223700 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.223469    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:19:30.225790 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.225759    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb53683e-628a-4184-9360-1ea2c224534f-metrics-tls\") pod \"dns-default-9mc89\" (UID: \"eb53683e-628a-4184-9360-1ea2c224534f\") " pod="openshift-dns/dns-default-9mc89"
May 28 03:19:30.225901 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.225849    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a1688cd-5f8c-4782-9652-9f0ab1e0206d-cert\") pod \"ingress-canary-b4qwv\" (UID: \"8a1688cd-5f8c-4782-9652-9f0ab1e0206d\") " pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:19:30.475828 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.475736    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-krlmb\""
May 28 03:19:30.476450 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.476432    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vsmkp\""
May 28 03:19:30.484114 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.484096    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b4qwv"
May 28 03:19:30.484215 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.484126    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9mc89"
May 28 03:19:30.635166 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.635130    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b4qwv"]
May 28 03:19:30.638864 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:30.638834    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1688cd_5f8c_4782_9652_9f0ab1e0206d.slice/crio-a44eec4171620a0ce367f96628e4d3fb62743b5b9faa918776792f63eb63df78 WatchSource:0}: Error finding container a44eec4171620a0ce367f96628e4d3fb62743b5b9faa918776792f63eb63df78: Status 404 returned error can't find the container with id a44eec4171620a0ce367f96628e4d3fb62743b5b9faa918776792f63eb63df78
May 28 03:19:30.647997 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:30.647972    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9mc89"]
May 28 03:19:30.651093 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:30.651068    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb53683e_628a_4184_9360_1ea2c224534f.slice/crio-dbb539fb825bf8843b4d92d5bf015515e9bdf14038f5ab6cb98886ac37a68386 WatchSource:0}: Error finding container dbb539fb825bf8843b4d92d5bf015515e9bdf14038f5ab6cb98886ac37a68386: Status 404 returned error can't find the container with id dbb539fb825bf8843b4d92d5bf015515e9bdf14038f5ab6cb98886ac37a68386
May 28 03:19:31.396084 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:31.396030    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b4qwv" event={"ID":"8a1688cd-5f8c-4782-9652-9f0ab1e0206d","Type":"ContainerStarted","Data":"a44eec4171620a0ce367f96628e4d3fb62743b5b9faa918776792f63eb63df78"}
May 28 03:19:31.397201 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:31.397170    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9mc89" event={"ID":"eb53683e-628a-4184-9360-1ea2c224534f","Type":"ContainerStarted","Data":"dbb539fb825bf8843b4d92d5bf015515e9bdf14038f5ab6cb98886ac37a68386"}
May 28 03:19:31.862134 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:31.862095    2569 scope.go:117] "RemoveContainer" containerID="f7c9fe2e92b36c5ff02bbbfa928f325ee355f4221af10f2ff8926289ddf16a49"
May 28 03:19:32.405124 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:32.405096    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:19:32.405303 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:32.405196    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4" event={"ID":"04dfd913-171c-498e-b7ae-d73e936b328f","Type":"ContainerStarted","Data":"0868022fd661202468e68da907a60553158fc1ccb9b64e9e59e55b9b5f2fe9f4"}
May 28 03:19:32.405636 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:32.405616    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:32.425234 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:32.425180    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4" podStartSLOduration=20.021865456 podStartE2EDuration="22.425162141s" podCreationTimestamp="2026-05-28 03:19:10 +0000 UTC" firstStartedPulling="2026-05-28 03:19:11.054535637 +0000 UTC m=+142.770056307" lastFinishedPulling="2026-05-28 03:19:13.45783232 +0000 UTC m=+145.173352992" observedRunningTime="2026-05-28 03:19:32.423612376 +0000 UTC m=+164.139133060" watchObservedRunningTime="2026-05-28 03:19:32.425162141 +0000 UTC m=+164.140682844"
May 28 03:19:33.041613 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:33.041524    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-bqzm4"
May 28 03:19:33.408818 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:33.408780    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b4qwv" event={"ID":"8a1688cd-5f8c-4782-9652-9f0ab1e0206d","Type":"ContainerStarted","Data":"3592b83919fb024784888eac808f23c48e2a7906f92c7f52c66b757e8577ec93"}
May 28 03:19:33.410400 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:33.410374    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9mc89" event={"ID":"eb53683e-628a-4184-9360-1ea2c224534f","Type":"ContainerStarted","Data":"4dcfe15b89710a436a50367850fa6c6e6d3effe96fab0291e750f08d63c10f00"}
May 28 03:19:33.410400 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:33.410403    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9mc89" event={"ID":"eb53683e-628a-4184-9360-1ea2c224534f","Type":"ContainerStarted","Data":"4163a285cf087f64e79f66a1100615863b78650b24bee6dc4bce69391f63de95"}
May 28 03:19:33.410601 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:33.410583    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9mc89"
May 28 03:19:33.427584 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:33.427540    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b4qwv" podStartSLOduration=129.564983861 podStartE2EDuration="2m11.427529185s" podCreationTimestamp="2026-05-28 03:17:22 +0000 UTC" firstStartedPulling="2026-05-28 03:19:30.641246388 +0000 UTC m=+162.356767061" lastFinishedPulling="2026-05-28 03:19:32.503791705 +0000 UTC m=+164.219312385" observedRunningTime="2026-05-28 03:19:33.426962674 +0000 UTC m=+165.142483366" watchObservedRunningTime="2026-05-28 03:19:33.427529185 +0000 UTC m=+165.143049911"
May 28 03:19:33.445038 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:33.444991    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-9mc89" podStartSLOduration=129.638378704 podStartE2EDuration="2m11.444978075s" podCreationTimestamp="2026-05-28 03:17:22 +0000 UTC" firstStartedPulling="2026-05-28 03:19:30.652862111 +0000 UTC m=+162.368382794" lastFinishedPulling="2026-05-28 03:19:32.459461496 +0000 UTC m=+164.174982165" observedRunningTime="2026-05-28 03:19:33.443928428 +0000 UTC m=+165.159449130" watchObservedRunningTime="2026-05-28 03:19:33.444978075 +0000 UTC m=+165.160498766"
May 28 03:19:38.863076 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:38.863041    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:19:40.968183 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:40.968153    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-78rv9"]
May 28 03:19:40.971187 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:40.971163    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:40.974662 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:40.974639    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 28 03:19:40.974800 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:40.974689    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 28 03:19:40.974800 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:40.974650    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 28 03:19:40.974800 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:40.974640    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xkttp\""
May 28 03:19:40.974800 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:40.974762    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 28 03:19:41.008836 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.008804    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-data-volume\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.009000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.008841    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.009000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.008871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-crio-socket\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.009000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.008934    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndxqq\" (UniqueName: \"kubernetes.io/projected/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-kube-api-access-ndxqq\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.009000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.008963    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.029576 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.029539    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-78rv9"]
May 28 03:19:41.041165 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.041137    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f758b8f98-x985d"]
May 28 03:19:41.041374 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:41.041336    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6f758b8f98-x985d" podUID="152b3248-ef2a-4e6f-924b-609074c896a2"
May 28 03:19:41.052925 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.052891    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-rgqh9"]
May 28 03:19:41.055660 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.055639    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-rgqh9"
May 28 03:19:41.058201 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.058182    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 28 03:19:41.058311 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.058222    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9ldf2\""
May 28 03:19:41.058644 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.058629    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 28 03:19:41.071740 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.071704    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-rgqh9"]
May 28 03:19:41.109687 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.109645    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-data-volume\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.109687 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.109685    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.109945 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.109726    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-crio-socket\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.109945 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.109762    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngf6t\" (UniqueName: \"kubernetes.io/projected/b3422cfc-ac5d-49d9-9f8e-0f5b73d68b30-kube-api-access-ngf6t\") pod \"downloads-57bd96b888-rgqh9\" (UID: \"b3422cfc-ac5d-49d9-9f8e-0f5b73d68b30\") " pod="openshift-console/downloads-57bd96b888-rgqh9"
May 28 03:19:41.109945 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.109804    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ndxqq\" (UniqueName: \"kubernetes.io/projected/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-kube-api-access-ndxqq\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.109945 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.109849    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.109945 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.109876    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-crio-socket\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.110178 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.110046    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-data-volume\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.110298 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.110278    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.112245 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.112222    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.119578 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.119556    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndxqq\" (UniqueName: \"kubernetes.io/projected/ddc3070d-dd0b-4926-9bb7-d4ed87f035de-kube-api-access-ndxqq\") pod \"insights-runtime-extractor-78rv9\" (UID: \"ddc3070d-dd0b-4926-9bb7-d4ed87f035de\") " pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.160087 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.160049    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84594f8d74-nlpnd"]
May 28 03:19:41.163109 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.163089    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.176030 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.176005    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84594f8d74-nlpnd"]
May 28 03:19:41.210437 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.210399    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkfq\" (UniqueName: \"kubernetes.io/projected/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-kube-api-access-bvkfq\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.210595 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.210451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-trusted-ca\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.210595 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.210520    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-ca-trust-extracted\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.210595 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.210564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-bound-sa-token\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.210595 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.210589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-image-registry-private-configuration\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.210756 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.210635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-registry-tls\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.210756 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.210674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-registry-certificates\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.210756 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.210697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-installation-pull-secrets\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.210756 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.210721    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ngf6t\" (UniqueName: \"kubernetes.io/projected/b3422cfc-ac5d-49d9-9f8e-0f5b73d68b30-kube-api-access-ngf6t\") pod \"downloads-57bd96b888-rgqh9\" (UID: \"b3422cfc-ac5d-49d9-9f8e-0f5b73d68b30\") " pod="openshift-console/downloads-57bd96b888-rgqh9"
May 28 03:19:41.224666 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.224600    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngf6t\" (UniqueName: \"kubernetes.io/projected/b3422cfc-ac5d-49d9-9f8e-0f5b73d68b30-kube-api-access-ngf6t\") pod \"downloads-57bd96b888-rgqh9\" (UID: \"b3422cfc-ac5d-49d9-9f8e-0f5b73d68b30\") " pod="openshift-console/downloads-57bd96b888-rgqh9"
May 28 03:19:41.281490 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.281460    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-78rv9"
May 28 03:19:41.311674 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.311642    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-installation-pull-secrets\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.311842 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.311703    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkfq\" (UniqueName: \"kubernetes.io/projected/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-kube-api-access-bvkfq\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.311842 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.311729    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-trusted-ca\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.311842 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.311750    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-ca-trust-extracted\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.311842 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.311783    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-bound-sa-token\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.312032 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.311919    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-image-registry-private-configuration\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.312032 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.311968    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-registry-tls\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.312032 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.312002    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-registry-certificates\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.312220 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.312199    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-ca-trust-extracted\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.312771 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.312722    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-registry-certificates\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.313085 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.313057    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-trusted-ca\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.314940 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.314915    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-registry-tls\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.315113 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.315094    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-image-registry-private-configuration\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.315163 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.315149    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-installation-pull-secrets\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.322634 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.322605    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-bound-sa-token\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.322758 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.322744    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkfq\" (UniqueName: \"kubernetes.io/projected/2a3070b1-0be1-4c5d-8014-ed51a22d0e18-kube-api-access-bvkfq\") pod \"image-registry-84594f8d74-nlpnd\" (UID: \"2a3070b1-0be1-4c5d-8014-ed51a22d0e18\") " pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.364698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.364668    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-rgqh9"
May 28 03:19:41.423146 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.423092    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-78rv9"]
May 28 03:19:41.428475 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:41.428447    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc3070d_dd0b_4926_9bb7_d4ed87f035de.slice/crio-f741ea0771f9d6885e3f8dff57286abb4169969cf3fd347185c59efcbc5241f5 WatchSource:0}: Error finding container f741ea0771f9d6885e3f8dff57286abb4169969cf3fd347185c59efcbc5241f5: Status 404 returned error can't find the container with id f741ea0771f9d6885e3f8dff57286abb4169969cf3fd347185c59efcbc5241f5
May 28 03:19:41.430514 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.430486    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:41.438657 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.438633    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:41.472318 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.472282    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:41.501200 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.501170    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-rgqh9"]
May 28 03:19:41.505047 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:41.504999    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3422cfc_ac5d_49d9_9f8e_0f5b73d68b30.slice/crio-ce13bc4dac4228064235c71c224e7c6e84a291d3cf0f6fb9f78e18b2d5a54d82 WatchSource:0}: Error finding container ce13bc4dac4228064235c71c224e7c6e84a291d3cf0f6fb9f78e18b2d5a54d82: Status 404 returned error can't find the container with id ce13bc4dac4228064235c71c224e7c6e84a291d3cf0f6fb9f78e18b2d5a54d82
May 28 03:19:41.513484 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.513457    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-image-registry-private-configuration\") pod \"152b3248-ef2a-4e6f-924b-609074c896a2\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") "
May 28 03:19:41.513606 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.513513    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-registry-certificates\") pod \"152b3248-ef2a-4e6f-924b-609074c896a2\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") "
May 28 03:19:41.513606 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.513542    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-trusted-ca\") pod \"152b3248-ef2a-4e6f-924b-609074c896a2\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") "
May 28 03:19:41.513606 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.513589    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-bound-sa-token\") pod \"152b3248-ef2a-4e6f-924b-609074c896a2\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") "
May 28 03:19:41.513767 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.513609    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152b3248-ef2a-4e6f-924b-609074c896a2-ca-trust-extracted\") pod \"152b3248-ef2a-4e6f-924b-609074c896a2\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") "
May 28 03:19:41.513767 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.513650    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-installation-pull-secrets\") pod \"152b3248-ef2a-4e6f-924b-609074c896a2\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") "
May 28 03:19:41.513767 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.513681    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-kube-api-access-8xj66\") pod \"152b3248-ef2a-4e6f-924b-609074c896a2\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") "
May 28 03:19:41.514082 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.514051    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152b3248-ef2a-4e6f-924b-609074c896a2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "152b3248-ef2a-4e6f-924b-609074c896a2" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 03:19:41.514257 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.514073    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "152b3248-ef2a-4e6f-924b-609074c896a2" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:19:41.514427 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.514107    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "152b3248-ef2a-4e6f-924b-609074c896a2" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:19:41.516331 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.516299    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-kube-api-access-8xj66" (OuterVolumeSpecName: "kube-api-access-8xj66") pod "152b3248-ef2a-4e6f-924b-609074c896a2" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2"). InnerVolumeSpecName "kube-api-access-8xj66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:19:41.516450 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.516326    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "152b3248-ef2a-4e6f-924b-609074c896a2" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:19:41.516450 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.516335    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "152b3248-ef2a-4e6f-924b-609074c896a2" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:19:41.516450 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.516335    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "152b3248-ef2a-4e6f-924b-609074c896a2" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:19:41.611120 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.611087    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84594f8d74-nlpnd"]
May 28 03:19:41.614516 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:41.614481    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3070b1_0be1_4c5d_8014_ed51a22d0e18.slice/crio-4d6d59e6579756fbbd82b850518b15d1804b7a419b9a1589043fe66170fb75ae WatchSource:0}: Error finding container 4d6d59e6579756fbbd82b850518b15d1804b7a419b9a1589043fe66170fb75ae: Status 404 returned error can't find the container with id 4d6d59e6579756fbbd82b850518b15d1804b7a419b9a1589043fe66170fb75ae
May 28 03:19:41.614593 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.614510    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-installation-pull-secrets\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:19:41.614593 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.614536    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-kube-api-access-8xj66\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:19:41.614593 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.614555    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152b3248-ef2a-4e6f-924b-609074c896a2-image-registry-private-configuration\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:19:41.614593 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.614571    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-registry-certificates\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:19:41.614593 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.614585    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152b3248-ef2a-4e6f-924b-609074c896a2-trusted-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:19:41.614750 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.614600    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-bound-sa-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:19:41.614750 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:41.614613    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152b3248-ef2a-4e6f-924b-609074c896a2-ca-trust-extracted\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:19:42.421294 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.421213    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:42.423700 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.423673    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"image-registry-6f758b8f98-x985d\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") " pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:42.435182 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.435156    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-78rv9" event={"ID":"ddc3070d-dd0b-4926-9bb7-d4ed87f035de","Type":"ContainerStarted","Data":"9ec42339605fc492e4ccd4c00e0f84a677da5a0a8b5839a510270701bcad0f52"}
May 28 03:19:42.435308 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.435191    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-78rv9" event={"ID":"ddc3070d-dd0b-4926-9bb7-d4ed87f035de","Type":"ContainerStarted","Data":"423301085ba6f630715b919487e75880ab13ecd3b000d7dd5e5e47dc6dc883c6"}
May 28 03:19:42.435308 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.435202    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-78rv9" event={"ID":"ddc3070d-dd0b-4926-9bb7-d4ed87f035de","Type":"ContainerStarted","Data":"f741ea0771f9d6885e3f8dff57286abb4169969cf3fd347185c59efcbc5241f5"}
May 28 03:19:42.436302 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.436279    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84594f8d74-nlpnd" event={"ID":"2a3070b1-0be1-4c5d-8014-ed51a22d0e18","Type":"ContainerStarted","Data":"759a9cfe7d434f164cb3eaa37f53fd8a8029ea95a2a51cf567850432dcb37901"}
May 28 03:19:42.436384 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.436310    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84594f8d74-nlpnd" event={"ID":"2a3070b1-0be1-4c5d-8014-ed51a22d0e18","Type":"ContainerStarted","Data":"4d6d59e6579756fbbd82b850518b15d1804b7a419b9a1589043fe66170fb75ae"}
May 28 03:19:42.436384 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.436373    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:19:42.437280 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.437265    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f758b8f98-x985d"
May 28 03:19:42.437385 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.437263    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-rgqh9" event={"ID":"b3422cfc-ac5d-49d9-9f8e-0f5b73d68b30","Type":"ContainerStarted","Data":"ce13bc4dac4228064235c71c224e7c6e84a291d3cf0f6fb9f78e18b2d5a54d82"}
May 28 03:19:42.459057 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.459015    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84594f8d74-nlpnd" podStartSLOduration=1.459001898 podStartE2EDuration="1.459001898s" podCreationTimestamp="2026-05-28 03:19:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 03:19:42.458110379 +0000 UTC m=+174.173631071" watchObservedRunningTime="2026-05-28 03:19:42.459001898 +0000 UTC m=+174.174522590"
May 28 03:19:42.521794 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.521757    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") pod \"152b3248-ef2a-4e6f-924b-609074c896a2\" (UID: \"152b3248-ef2a-4e6f-924b-609074c896a2\") "
May 28 03:19:42.524214 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.524180    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "152b3248-ef2a-4e6f-924b-609074c896a2" (UID: "152b3248-ef2a-4e6f-924b-609074c896a2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:19:42.622785 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.622741    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152b3248-ef2a-4e6f-924b-609074c896a2-registry-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:19:42.771952 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.771920    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f758b8f98-x985d"]
May 28 03:19:42.777598 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.777548    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f758b8f98-x985d"]
May 28 03:19:42.865638 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:42.865605    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="152b3248-ef2a-4e6f-924b-609074c896a2" path="/var/lib/kubelet/pods/152b3248-ef2a-4e6f-924b-609074c896a2/volumes"
May 28 03:19:43.415482 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:43.415456    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9mc89"
May 28 03:19:44.446771 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:44.446736    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-78rv9" event={"ID":"ddc3070d-dd0b-4926-9bb7-d4ed87f035de","Type":"ContainerStarted","Data":"4b0e1563a904e79380d937f4c41047e3179aeb9fa7094d772f472aa33675e5c3"}
May 28 03:19:44.470446 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:44.470394    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-78rv9" podStartSLOduration=1.918258493 podStartE2EDuration="4.470379685s" podCreationTimestamp="2026-05-28 03:19:40 +0000 UTC" firstStartedPulling="2026-05-28 03:19:41.504994089 +0000 UTC m=+173.220514764" lastFinishedPulling="2026-05-28 03:19:44.057115286 +0000 UTC m=+175.772635956" observedRunningTime="2026-05-28 03:19:44.470170096 +0000 UTC m=+176.185690786" watchObservedRunningTime="2026-05-28 03:19:44.470379685 +0000 UTC m=+176.185900378"
May 28 03:19:49.108148 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.108106    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"]
May 28 03:19:49.112377 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.112353    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.114724 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.114698    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vpsfh\""
May 28 03:19:49.115109 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.115071    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 28 03:19:49.115109 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.115096    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 28 03:19:49.115267 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.115127    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 28 03:19:49.115267 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.115073    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 28 03:19:49.115648 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.115626    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 28 03:19:49.121195 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.121165    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"]
May 28 03:19:49.177332 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.177290    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rk9m\" (UniqueName: \"kubernetes.io/projected/c2f27497-7ee2-46ee-b1b3-95eda95c7433-kube-api-access-5rk9m\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.177521 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.177402    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f27497-7ee2-46ee-b1b3-95eda95c7433-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.177521 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.177444    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f27497-7ee2-46ee-b1b3-95eda95c7433-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.177521 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.177468    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f27497-7ee2-46ee-b1b3-95eda95c7433-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.278519 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.278481    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f27497-7ee2-46ee-b1b3-95eda95c7433-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.278519 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.278523    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f27497-7ee2-46ee-b1b3-95eda95c7433-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.278727 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.278554    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5rk9m\" (UniqueName: \"kubernetes.io/projected/c2f27497-7ee2-46ee-b1b3-95eda95c7433-kube-api-access-5rk9m\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.278727 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.278597    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f27497-7ee2-46ee-b1b3-95eda95c7433-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.279281 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.279174    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f27497-7ee2-46ee-b1b3-95eda95c7433-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.281668 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.281638    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2f27497-7ee2-46ee-b1b3-95eda95c7433-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.281771 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.281674    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f27497-7ee2-46ee-b1b3-95eda95c7433-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.288797 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.288778    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rk9m\" (UniqueName: \"kubernetes.io/projected/c2f27497-7ee2-46ee-b1b3-95eda95c7433-kube-api-access-5rk9m\") pod \"prometheus-operator-7779bbdd78-m8lzx\" (UID: \"c2f27497-7ee2-46ee-b1b3-95eda95c7433\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.424537 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.424453    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"
May 28 03:19:49.556560 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:49.556527    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx"]
May 28 03:19:49.559277 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:49.559249    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f27497_7ee2_46ee_b1b3_95eda95c7433.slice/crio-db2d9fae9f65a4971bba5431e3af7844ceec77c12cf0eeccf69c9c5ed512662e WatchSource:0}: Error finding container db2d9fae9f65a4971bba5431e3af7844ceec77c12cf0eeccf69c9c5ed512662e: Status 404 returned error can't find the container with id db2d9fae9f65a4971bba5431e3af7844ceec77c12cf0eeccf69c9c5ed512662e
May 28 03:19:50.462789 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:50.462749    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx" event={"ID":"c2f27497-7ee2-46ee-b1b3-95eda95c7433","Type":"ContainerStarted","Data":"db2d9fae9f65a4971bba5431e3af7844ceec77c12cf0eeccf69c9c5ed512662e"}
May 28 03:19:51.466785 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:51.466747    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx" event={"ID":"c2f27497-7ee2-46ee-b1b3-95eda95c7433","Type":"ContainerStarted","Data":"57b81dac93c537a654757737f3369ebff9e616bff758e750ad177109b235c835"}
May 28 03:19:51.466785 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:51.466785    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx" event={"ID":"c2f27497-7ee2-46ee-b1b3-95eda95c7433","Type":"ContainerStarted","Data":"a28df77e20f39bc7dbe45c575319ed86653c46c2251d17d3d0d996880699fd62"}
May 28 03:19:51.487234 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:51.487186    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-m8lzx" podStartSLOduration=0.982101999 podStartE2EDuration="2.487172112s" podCreationTimestamp="2026-05-28 03:19:49 +0000 UTC" firstStartedPulling="2026-05-28 03:19:49.561092716 +0000 UTC m=+181.276613399" lastFinishedPulling="2026-05-28 03:19:51.066162839 +0000 UTC m=+182.781683512" observedRunningTime="2026-05-28 03:19:51.486134625 +0000 UTC m=+183.201655321" watchObservedRunningTime="2026-05-28 03:19:51.487172112 +0000 UTC m=+183.202692805"
May 28 03:19:53.459859 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.459820    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"]
May 28 03:19:53.466077 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.466054    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.469573 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.469542    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 28 03:19:53.469875 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.469860    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z7hk4\""
May 28 03:19:53.470666 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.470646    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 28 03:19:53.486082 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.486054    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"]
May 28 03:19:53.491189 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.491164    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q4zm5"]
May 28 03:19:53.495898 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.495876    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.499834 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.499783    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 28 03:19:53.502931 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.502909    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 28 03:19:53.503440 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.503422    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7qxsc\""
May 28 03:19:53.503539 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.503433    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 28 03:19:53.617219 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-textfile\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.617433 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f723a40-b86a-4c90-a0ae-b027eb38884b-sys\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.617433 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f723a40-b86a-4c90-a0ae-b027eb38884b-metrics-client-ca\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.617433 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhlv\" (UniqueName: \"kubernetes.io/projected/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-kube-api-access-vjhlv\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.617571 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617428    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.617571 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-wtmp\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.617571 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617491    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.617571 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617522    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-tls\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.617571 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617561    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.617825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.617825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617620    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.617825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617646    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f723a40-b86a-4c90-a0ae-b027eb38884b-root\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.617825 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.617669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rl6m\" (UniqueName: \"kubernetes.io/projected/4f723a40-b86a-4c90-a0ae-b027eb38884b-kube-api-access-9rl6m\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.718959 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.718872    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-textfile\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719124 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719014    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f723a40-b86a-4c90-a0ae-b027eb38884b-sys\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719124 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719053    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f723a40-b86a-4c90-a0ae-b027eb38884b-metrics-client-ca\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719124 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719095    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhlv\" (UniqueName: \"kubernetes.io/projected/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-kube-api-access-vjhlv\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.719124 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719101    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f723a40-b86a-4c90-a0ae-b027eb38884b-sys\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719337 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719139    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.719337 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719165    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-wtmp\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719337 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719200    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719337 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719233    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-tls\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719337 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719274    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.719337 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719314    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719373    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-wtmp\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719384    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.719698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719427    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f723a40-b86a-4c90-a0ae-b027eb38884b-root\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719458    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9rl6m\" (UniqueName: \"kubernetes.io/projected/4f723a40-b86a-4c90-a0ae-b027eb38884b-kube-api-access-9rl6m\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719694    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f723a40-b86a-4c90-a0ae-b027eb38884b-metrics-client-ca\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.719951 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719768    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f723a40-b86a-4c90-a0ae-b027eb38884b-root\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.720014 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.719229    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-textfile\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.720307 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.720254    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.720440 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.720329    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.722140 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.722118    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.722513 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.722488    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.722614 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.722495    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.722614 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.722520    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f723a40-b86a-4c90-a0ae-b027eb38884b-node-exporter-tls\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.727774 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.727746    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rl6m\" (UniqueName: \"kubernetes.io/projected/4f723a40-b86a-4c90-a0ae-b027eb38884b-kube-api-access-9rl6m\") pod \"node-exporter-q4zm5\" (UID: \"4f723a40-b86a-4c90-a0ae-b027eb38884b\") " pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.728056 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.728028    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhlv\" (UniqueName: \"kubernetes.io/projected/545c3d5e-8f94-4b9f-a3a1-5478495dbee3-kube-api-access-vjhlv\") pod \"openshift-state-metrics-9c959559f-x8mdt\" (UID: \"545c3d5e-8f94-4b9f-a3a1-5478495dbee3\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.780792 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.780752    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"
May 28 03:19:53.808638 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.808605    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q4zm5"
May 28 03:19:53.816818 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:53.816786    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f723a40_b86a_4c90_a0ae_b027eb38884b.slice/crio-8901d31d158c04e2cde2aaa2f6c2fdeb13bfd65ca9c28b9654b1a0f0e352ea63 WatchSource:0}: Error finding container 8901d31d158c04e2cde2aaa2f6c2fdeb13bfd65ca9c28b9654b1a0f0e352ea63: Status 404 returned error can't find the container with id 8901d31d158c04e2cde2aaa2f6c2fdeb13bfd65ca9c28b9654b1a0f0e352ea63
May 28 03:19:53.943918 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:53.943880    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt"]
May 28 03:19:53.948082 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:53.948044    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545c3d5e_8f94_4b9f_a3a1_5478495dbee3.slice/crio-aca158f2d984d4295d5e5863055bf6eab687696f534c4ccd5b9decf1bc438940 WatchSource:0}: Error finding container aca158f2d984d4295d5e5863055bf6eab687696f534c4ccd5b9decf1bc438940: Status 404 returned error can't find the container with id aca158f2d984d4295d5e5863055bf6eab687696f534c4ccd5b9decf1bc438940
May 28 03:19:54.478485 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:54.478442    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt" event={"ID":"545c3d5e-8f94-4b9f-a3a1-5478495dbee3","Type":"ContainerStarted","Data":"83d9b11cd658a5147b4a28d1132c7d0c31fcf9a586c9f65257fa92f6fa4ceee1"}
May 28 03:19:54.478951 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:54.478495    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt" event={"ID":"545c3d5e-8f94-4b9f-a3a1-5478495dbee3","Type":"ContainerStarted","Data":"85c859babdc8b1d34ccebdf0e0d1c764a55210b890884b1b10f9d1c393ea0e81"}
May 28 03:19:54.478951 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:54.478510    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt" event={"ID":"545c3d5e-8f94-4b9f-a3a1-5478495dbee3","Type":"ContainerStarted","Data":"aca158f2d984d4295d5e5863055bf6eab687696f534c4ccd5b9decf1bc438940"}
May 28 03:19:54.483740 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:54.483673    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4zm5" event={"ID":"4f723a40-b86a-4c90-a0ae-b027eb38884b","Type":"ContainerStarted","Data":"8901d31d158c04e2cde2aaa2f6c2fdeb13bfd65ca9c28b9654b1a0f0e352ea63"}
May 28 03:19:55.491921 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.491889    2569 generic.go:358] "Generic (PLEG): container finished" podID="4f723a40-b86a-4c90-a0ae-b027eb38884b" containerID="89e892255dc17fe89e7404fa4fad3d00cdadacc681e3174f51a8f8e6c0a5bc90" exitCode=0
May 28 03:19:55.492291 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.491946    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4zm5" event={"ID":"4f723a40-b86a-4c90-a0ae-b027eb38884b","Type":"ContainerDied","Data":"89e892255dc17fe89e7404fa4fad3d00cdadacc681e3174f51a8f8e6c0a5bc90"}
May 28 03:19:55.776780 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.776702    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7899f9f89-6m59r"]
May 28 03:19:55.780213 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.780188    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:55.785550 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.785514    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 28 03:19:55.786138 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.786116    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8lgv8\""
May 28 03:19:55.786246 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.786218    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 28 03:19:55.786465 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.786446    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 28 03:19:55.786569 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.786489    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8oc86s0k2sjvo\""
May 28 03:19:55.787298 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.787126    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 28 03:19:55.787729 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.787688    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 28 03:19:55.808824 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.808793    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7899f9f89-6m59r"]
May 28 03:19:55.937588 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.937556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcc4\" (UniqueName: \"kubernetes.io/projected/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-kube-api-access-8tcc4\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:55.937782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.937600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:55.937782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.937687    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:55.937782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.937717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:55.937782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.937744    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-tls\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:55.937782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.937768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:55.937972 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.937837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-grpc-tls\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:55.937972 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:55.937868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-metrics-client-ca\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.038879 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.038795    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.038879 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.038834    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.038879 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.038854    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-tls\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.038879 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.038871    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.039158 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.038898    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-grpc-tls\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.039158 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.038991    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-metrics-client-ca\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.039158 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.039076    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcc4\" (UniqueName: \"kubernetes.io/projected/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-kube-api-access-8tcc4\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.039158 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.039111    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.039834 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.039806    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-metrics-client-ca\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.041677 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.041622    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.041790 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.041739    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.042004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.041985    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.042077 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.042000    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-tls\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.042139 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.042099    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-grpc-tls\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.042139 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.042108    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.053184 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.053157    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcc4\" (UniqueName: \"kubernetes.io/projected/3d8fa66f-6d37-40d1-99a1-d53640bc50a6-kube-api-access-8tcc4\") pod \"thanos-querier-7899f9f89-6m59r\" (UID: \"3d8fa66f-6d37-40d1-99a1-d53640bc50a6\") " pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.091974 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.091944    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:19:56.244132 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.242933    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7899f9f89-6m59r"]
May 28 03:19:56.246968 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:56.246939    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8fa66f_6d37_40d1_99a1_d53640bc50a6.slice/crio-1073708cdb93cc5d7acf4bec5b50d12d0c1601dff22abbd027c1316f01b52e38 WatchSource:0}: Error finding container 1073708cdb93cc5d7acf4bec5b50d12d0c1601dff22abbd027c1316f01b52e38: Status 404 returned error can't find the container with id 1073708cdb93cc5d7acf4bec5b50d12d0c1601dff22abbd027c1316f01b52e38
May 28 03:19:56.386988 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.386953    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-65dcbf9b66-rsf9k"]
May 28 03:19:56.391446 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.391416    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.394182 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.393928    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-prlcr\""
May 28 03:19:56.394182 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.393951    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 28 03:19:56.394560 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.394389    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 28 03:19:56.394560 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.394426    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 28 03:19:56.394560 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.394465    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 28 03:19:56.394872 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.394852    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 28 03:19:56.399682 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.399659    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 28 03:19:56.409353 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.409313    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dcbf9b66-rsf9k"]
May 28 03:19:56.496679 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.496645    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt" event={"ID":"545c3d5e-8f94-4b9f-a3a1-5478495dbee3","Type":"ContainerStarted","Data":"0424356fd37f3053b91e4b2d3222c637f0cb995dc9878f56ebf7fb169fbb428f"}
May 28 03:19:56.497751 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.497725    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r" event={"ID":"3d8fa66f-6d37-40d1-99a1-d53640bc50a6","Type":"ContainerStarted","Data":"1073708cdb93cc5d7acf4bec5b50d12d0c1601dff22abbd027c1316f01b52e38"}
May 28 03:19:56.499412 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.499393    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4zm5" event={"ID":"4f723a40-b86a-4c90-a0ae-b027eb38884b","Type":"ContainerStarted","Data":"706508ce239d22da0fa7f049a5fb29855a12bb788d20a5f8fc2d3b726e68d0fd"}
May 28 03:19:56.499484 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.499417    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4zm5" event={"ID":"4f723a40-b86a-4c90-a0ae-b027eb38884b","Type":"ContainerStarted","Data":"ea377bc5a7e45b96bbaf4507f29dba7338f8b0edf569eb880f15f9fb3e2f6475"}
May 28 03:19:56.523101 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.523052    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8mdt" podStartSLOduration=2.179774457 podStartE2EDuration="3.523038143s" podCreationTimestamp="2026-05-28 03:19:53 +0000 UTC" firstStartedPulling="2026-05-28 03:19:54.089560599 +0000 UTC m=+185.805081285" lastFinishedPulling="2026-05-28 03:19:55.432824302 +0000 UTC m=+187.148344971" observedRunningTime="2026-05-28 03:19:56.521479123 +0000 UTC m=+188.236999828" watchObservedRunningTime="2026-05-28 03:19:56.523038143 +0000 UTC m=+188.238558834"
May 28 03:19:56.546165 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.546132    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-service-ca\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.546165 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.546169    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27qdl\" (UniqueName: \"kubernetes.io/projected/ca225e1d-b41a-4c8c-9121-53becbc73fc1-kube-api-access-27qdl\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.546397 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.546245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-trusted-ca-bundle\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.546464 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.546371    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-serving-cert\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.546520 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.546484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-config\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.546655 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.546629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-oauth-config\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.546722 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.546687    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-oauth-serving-cert\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.557041 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.556982    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q4zm5" podStartSLOduration=2.446423521 podStartE2EDuration="3.556956558s" podCreationTimestamp="2026-05-28 03:19:53 +0000 UTC" firstStartedPulling="2026-05-28 03:19:53.818708408 +0000 UTC m=+185.534229081" lastFinishedPulling="2026-05-28 03:19:54.929241447 +0000 UTC m=+186.644762118" observedRunningTime="2026-05-28 03:19:56.555729449 +0000 UTC m=+188.271250154" watchObservedRunningTime="2026-05-28 03:19:56.556956558 +0000 UTC m=+188.272477251"
May 28 03:19:56.647633 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.647537    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-oauth-serving-cert\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.647633 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.647616    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-service-ca\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.647850 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.647644    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-27qdl\" (UniqueName: \"kubernetes.io/projected/ca225e1d-b41a-4c8c-9121-53becbc73fc1-kube-api-access-27qdl\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.647850 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.647721    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-trusted-ca-bundle\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.647850 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.647751    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-serving-cert\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.647850 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.647815    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-config\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.648042 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.647934    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-oauth-config\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.648645 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.648274    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-oauth-serving-cert\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.649387 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.649267    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-config\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.649549 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.649526    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-trusted-ca-bundle\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.649760 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.649740    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-service-ca\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.651505 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.651480    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-serving-cert\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.651589 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.651545    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-oauth-config\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.657315 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.657291    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27qdl\" (UniqueName: \"kubernetes.io/projected/ca225e1d-b41a-4c8c-9121-53becbc73fc1-kube-api-access-27qdl\") pod \"console-65dcbf9b66-rsf9k\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") " pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.704631 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.704591    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:19:56.876780 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:56.876742    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dcbf9b66-rsf9k"]
May 28 03:19:56.882494 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:19:56.882453    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca225e1d_b41a_4c8c_9121_53becbc73fc1.slice/crio-42f7cde63aca9c5b263a16cd979d7b490241c4a1539ffdbc9ebbdfaa897a3d9a WatchSource:0}: Error finding container 42f7cde63aca9c5b263a16cd979d7b490241c4a1539ffdbc9ebbdfaa897a3d9a: Status 404 returned error can't find the container with id 42f7cde63aca9c5b263a16cd979d7b490241c4a1539ffdbc9ebbdfaa897a3d9a
May 28 03:19:57.506053 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:57.505959    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dcbf9b66-rsf9k" event={"ID":"ca225e1d-b41a-4c8c-9121-53becbc73fc1","Type":"ContainerStarted","Data":"42f7cde63aca9c5b263a16cd979d7b490241c4a1539ffdbc9ebbdfaa897a3d9a"}
May 28 03:19:58.250310 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:58.249810    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"]
May 28 03:19:58.254461 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:58.253967    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"
May 28 03:19:58.256868 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:58.256537    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-7mfr8\""
May 28 03:19:58.256868 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:58.256754    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 28 03:19:58.267626 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:58.266861    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"]
May 28 03:19:58.366826 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:58.366748    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40bbbf19-4e88-423f-8f29-32c627762ffd-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-sbjxf\" (UID: \"40bbbf19-4e88-423f-8f29-32c627762ffd\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"
May 28 03:19:58.467703 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:58.467657    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40bbbf19-4e88-423f-8f29-32c627762ffd-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-sbjxf\" (UID: \"40bbbf19-4e88-423f-8f29-32c627762ffd\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"
May 28 03:19:58.467915 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:58.467879    2569 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 28 03:19:58.467986 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:19:58.467952    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40bbbf19-4e88-423f-8f29-32c627762ffd-monitoring-plugin-cert podName:40bbbf19-4e88-423f-8f29-32c627762ffd nodeName:}" failed. No retries permitted until 2026-05-28 03:19:58.967932624 +0000 UTC m=+190.683453316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/40bbbf19-4e88-423f-8f29-32c627762ffd-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-sbjxf" (UID: "40bbbf19-4e88-423f-8f29-32c627762ffd") : secret "monitoring-plugin-cert" not found
May 28 03:19:58.972590 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:58.972548    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40bbbf19-4e88-423f-8f29-32c627762ffd-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-sbjxf\" (UID: \"40bbbf19-4e88-423f-8f29-32c627762ffd\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"
May 28 03:19:58.975497 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:58.975467    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40bbbf19-4e88-423f-8f29-32c627762ffd-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-sbjxf\" (UID: \"40bbbf19-4e88-423f-8f29-32c627762ffd\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"
May 28 03:19:59.171179 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:19:59.171143    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"
May 28 03:20:00.020271 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.019518    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 03:20:00.024392 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.024326    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.027814 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.027781    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 28 03:20:00.027956 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.027850    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 28 03:20:00.028016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.027969    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7b9v9a1fougje\""
May 28 03:20:00.028079 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.028039    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 28 03:20:00.028967 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.028641    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n2f9h\""
May 28 03:20:00.028967 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.028695    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 28 03:20:00.028967 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.028711    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 28 03:20:00.028967 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.028830    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 28 03:20:00.028967 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.028836    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 28 03:20:00.028967 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.028925    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 28 03:20:00.028967 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.028944    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 28 03:20:00.029719 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.029681    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 28 03:20:00.030027 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.029904    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 28 03:20:00.030027 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.029960    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 28 03:20:00.032154 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.032133    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 28 03:20:00.038336 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.038312    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 03:20:00.184544 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184504    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.184740 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184561    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.184740 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.184740 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184687    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.184740 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184704    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.184740 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-web-config\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.184740 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185025 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184770    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhtr\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-kube-api-access-hfhtr\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185025 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185025 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184854    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185025 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185025 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184932    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185025 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-config-out\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185025 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.184994    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185025 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.185024    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185307 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.185048    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185307 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.185073    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.185307 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.185119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-config\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286671 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286586    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286671 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286635    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286671 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286664    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286927 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286693    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-config\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286927 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286740    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286927 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286772    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286927 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286800    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286927 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286857    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286927 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286882    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286927 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286904    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-web-config\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.286927 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286928    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.287330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286953    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhtr\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-kube-api-access-hfhtr\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.287330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.286994    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.287330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.287022    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.287330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.287048    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.287330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.287079    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.287330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.287114    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-config-out\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.287330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.287138    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.289471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.288305    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.289471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.288622    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.289471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.289079    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.289471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.289313    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.294511 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.293490    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.294511 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.293750    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-web-config\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.294511 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.293797    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.294511 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.294195    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.295257 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.295179    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.295372 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.295299    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.295923 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.295881    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-config\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.296161 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.296112    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.297473 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.297430    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-config-out\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.297574 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.297470    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.298147 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.298121    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.298627 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.298580    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.299650 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.299627    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.302739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.302716    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhtr\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-kube-api-access-hfhtr\") pod \"prometheus-k8s-0\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:00.340067 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:00.340025    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:01.155861 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:01.155821    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dcbf9b66-rsf9k"]
May 28 03:20:03.446675 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:03.446644    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84594f8d74-nlpnd"
May 28 03:20:07.029928 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.029898    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 03:20:07.033307 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:20:07.033279    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02315d19_0379_410e_a14f_89f5a6af1e53.slice/crio-a879065360d6d95d653f632b901d1e891b0eb6599f8be8578ede1e39938381d1 WatchSource:0}: Error finding container a879065360d6d95d653f632b901d1e891b0eb6599f8be8578ede1e39938381d1: Status 404 returned error can't find the container with id a879065360d6d95d653f632b901d1e891b0eb6599f8be8578ede1e39938381d1
May 28 03:20:07.055163 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.055112    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"]
May 28 03:20:07.071451 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:20:07.071423    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bbbf19_4e88_423f_8f29_32c627762ffd.slice/crio-c6c3bd7fa790fad3b72b990f4c156aa21460a1160cfcf9206850e4ee66d57263 WatchSource:0}: Error finding container c6c3bd7fa790fad3b72b990f4c156aa21460a1160cfcf9206850e4ee66d57263: Status 404 returned error can't find the container with id c6c3bd7fa790fad3b72b990f4c156aa21460a1160cfcf9206850e4ee66d57263
May 28 03:20:07.543441 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.543393    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf" event={"ID":"40bbbf19-4e88-423f-8f29-32c627762ffd","Type":"ContainerStarted","Data":"c6c3bd7fa790fad3b72b990f4c156aa21460a1160cfcf9206850e4ee66d57263"}
May 28 03:20:07.545207 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.544973    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerStarted","Data":"a879065360d6d95d653f632b901d1e891b0eb6599f8be8578ede1e39938381d1"}
May 28 03:20:07.552611 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.551184    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-rgqh9" event={"ID":"b3422cfc-ac5d-49d9-9f8e-0f5b73d68b30","Type":"ContainerStarted","Data":"9295b5f537041242905678c15b6e0826268be55886412d1aea65e6c51b3dfed7"}
May 28 03:20:07.552611 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.551825    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-rgqh9"
May 28 03:20:07.556454 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.556405    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r" event={"ID":"3d8fa66f-6d37-40d1-99a1-d53640bc50a6","Type":"ContainerStarted","Data":"2384e4c3cc8c6044b1605cd5fabb3c75e9fbd528acf75dc85eb1e30ca5a35fce"}
May 28 03:20:07.556454 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.556441    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r" event={"ID":"3d8fa66f-6d37-40d1-99a1-d53640bc50a6","Type":"ContainerStarted","Data":"eef4bbf3705d5af4aa0682b591dbb2e3e40762b944a6aab8c4823a2f97266c3d"}
May 28 03:20:07.556454 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.556454    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r" event={"ID":"3d8fa66f-6d37-40d1-99a1-d53640bc50a6","Type":"ContainerStarted","Data":"84a8eb3f137b0104d0eb98465e9aa571791a262607b59c198b78cc7b9556fa56"}
May 28 03:20:07.558914 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.558638    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dcbf9b66-rsf9k" event={"ID":"ca225e1d-b41a-4c8c-9121-53becbc73fc1","Type":"ContainerStarted","Data":"963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7"}
May 28 03:20:07.564587 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.564551    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-rgqh9"
May 28 03:20:07.569431 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.569378    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-rgqh9" podStartSLOduration=1.180953906 podStartE2EDuration="26.569359165s" podCreationTimestamp="2026-05-28 03:19:41 +0000 UTC" firstStartedPulling="2026-05-28 03:19:41.507073024 +0000 UTC m=+173.222593693" lastFinishedPulling="2026-05-28 03:20:06.895478265 +0000 UTC m=+198.610998952" observedRunningTime="2026-05-28 03:20:07.568063287 +0000 UTC m=+199.283583979" watchObservedRunningTime="2026-05-28 03:20:07.569359165 +0000 UTC m=+199.284879858"
May 28 03:20:07.603926 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:07.603798    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-65dcbf9b66-rsf9k" podStartSLOduration=1.645951546 podStartE2EDuration="11.603779799s" podCreationTimestamp="2026-05-28 03:19:56 +0000 UTC" firstStartedPulling="2026-05-28 03:19:56.88576777 +0000 UTC m=+188.601288455" lastFinishedPulling="2026-05-28 03:20:06.843596023 +0000 UTC m=+198.559116708" observedRunningTime="2026-05-28 03:20:07.58678348 +0000 UTC m=+199.302304172" watchObservedRunningTime="2026-05-28 03:20:07.603779799 +0000 UTC m=+199.319300492"
May 28 03:20:08.565875 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:08.565796    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r" event={"ID":"3d8fa66f-6d37-40d1-99a1-d53640bc50a6","Type":"ContainerStarted","Data":"a7bcb5584fe4b7c3664398697c8975ec61226f27e8ce4535e4752a2f9c1e9707"}
May 28 03:20:08.565875 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:08.565848    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r" event={"ID":"3d8fa66f-6d37-40d1-99a1-d53640bc50a6","Type":"ContainerStarted","Data":"62bd31e552299e76986c9e90cff73ca06e1a51e13c77fb6876a82779cac06c66"}
May 28 03:20:09.571899 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:09.571855    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r" event={"ID":"3d8fa66f-6d37-40d1-99a1-d53640bc50a6","Type":"ContainerStarted","Data":"5b022f1fc342e9d01599970e60277e6d4433e70543509b236c136e5066157474"}
May 28 03:20:09.572405 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:09.572058    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:20:09.573569 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:09.573540    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf" event={"ID":"40bbbf19-4e88-423f-8f29-32c627762ffd","Type":"ContainerStarted","Data":"99e841602745cbf73e9c57e958d01b71d32395f612e722ad9d8324ae8c1d85cb"}
May 28 03:20:09.573736 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:09.573707    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"
May 28 03:20:09.575174 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:09.575147    2569 generic.go:358] "Generic (PLEG): container finished" podID="02315d19-0379-410e-a14f-89f5a6af1e53" containerID="11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438" exitCode=0
May 28 03:20:09.575517 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:09.575490    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerDied","Data":"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"}
May 28 03:20:09.579660 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:09.579639    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf"
May 28 03:20:09.603018 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:09.602960    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r" podStartSLOduration=2.901909906 podStartE2EDuration="14.602943301s" podCreationTimestamp="2026-05-28 03:19:55 +0000 UTC" firstStartedPulling="2026-05-28 03:19:56.248794564 +0000 UTC m=+187.964315234" lastFinishedPulling="2026-05-28 03:20:07.949827945 +0000 UTC m=+199.665348629" observedRunningTime="2026-05-28 03:20:09.60181143 +0000 UTC m=+201.317332123" watchObservedRunningTime="2026-05-28 03:20:09.602943301 +0000 UTC m=+201.318463995"
May 28 03:20:09.622556 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:09.622461    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sbjxf" podStartSLOduration=9.324361784 podStartE2EDuration="11.622442352s" podCreationTimestamp="2026-05-28 03:19:58 +0000 UTC" firstStartedPulling="2026-05-28 03:20:07.07331335 +0000 UTC m=+198.788834023" lastFinishedPulling="2026-05-28 03:20:09.37139392 +0000 UTC m=+201.086914591" observedRunningTime="2026-05-28 03:20:09.620843907 +0000 UTC m=+201.336364599" watchObservedRunningTime="2026-05-28 03:20:09.622442352 +0000 UTC m=+201.337963046"
May 28 03:20:13.592121 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:13.592030    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerStarted","Data":"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"}
May 28 03:20:14.598849 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:14.598803    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerStarted","Data":"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"}
May 28 03:20:14.598849 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:14.598852    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerStarted","Data":"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"}
May 28 03:20:14.599479 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:14.598868    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerStarted","Data":"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"}
May 28 03:20:14.599479 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:14.598880    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerStarted","Data":"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"}
May 28 03:20:14.599479 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:14.598895    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerStarted","Data":"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"}
May 28 03:20:14.637230 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:14.637142    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=9.33085327 podStartE2EDuration="15.637123167s" podCreationTimestamp="2026-05-28 03:19:59 +0000 UTC" firstStartedPulling="2026-05-28 03:20:07.036048928 +0000 UTC m=+198.751569598" lastFinishedPulling="2026-05-28 03:20:13.342318812 +0000 UTC m=+205.057839495" observedRunningTime="2026-05-28 03:20:14.634119641 +0000 UTC m=+206.349640370" watchObservedRunningTime="2026-05-28 03:20:14.637123167 +0000 UTC m=+206.352643860"
May 28 03:20:15.340471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:15.340429    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:20:15.586732 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:15.586697    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7899f9f89-6m59r"
May 28 03:20:16.704824 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:16.704787    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:20:32.588606 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.588540    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-65dcbf9b66-rsf9k" podUID="ca225e1d-b41a-4c8c-9121-53becbc73fc1" containerName="console" containerID="cri-o://963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7" gracePeriod=15
May 28 03:20:32.838277 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.838254    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65dcbf9b66-rsf9k_ca225e1d-b41a-4c8c-9121-53becbc73fc1/console/0.log"
May 28 03:20:32.838413 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.838315    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:20:32.899908 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.899829    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-config\") pod \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") "
May 28 03:20:32.899908 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.899874    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27qdl\" (UniqueName: \"kubernetes.io/projected/ca225e1d-b41a-4c8c-9121-53becbc73fc1-kube-api-access-27qdl\") pod \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") "
May 28 03:20:32.899908 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.899902    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-trusted-ca-bundle\") pod \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") "
May 28 03:20:32.900160 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.899928    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-oauth-serving-cert\") pod \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") "
May 28 03:20:32.900160 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.899989    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-service-ca\") pod \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") "
May 28 03:20:32.900160 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.900015    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-serving-cert\") pod \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") "
May 28 03:20:32.900160 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.900062    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-oauth-config\") pod \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\" (UID: \"ca225e1d-b41a-4c8c-9121-53becbc73fc1\") "
May 28 03:20:32.900449 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.900398    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-service-ca" (OuterVolumeSpecName: "service-ca") pod "ca225e1d-b41a-4c8c-9121-53becbc73fc1" (UID: "ca225e1d-b41a-4c8c-9121-53becbc73fc1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:20:32.900449 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.900415    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-config" (OuterVolumeSpecName: "console-config") pod "ca225e1d-b41a-4c8c-9121-53becbc73fc1" (UID: "ca225e1d-b41a-4c8c-9121-53becbc73fc1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:20:32.900449 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.900423    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ca225e1d-b41a-4c8c-9121-53becbc73fc1" (UID: "ca225e1d-b41a-4c8c-9121-53becbc73fc1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:20:32.900631 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.900477    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ca225e1d-b41a-4c8c-9121-53becbc73fc1" (UID: "ca225e1d-b41a-4c8c-9121-53becbc73fc1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:20:32.902181 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.902159    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ca225e1d-b41a-4c8c-9121-53becbc73fc1" (UID: "ca225e1d-b41a-4c8c-9121-53becbc73fc1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:20:32.902247 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.902174    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca225e1d-b41a-4c8c-9121-53becbc73fc1-kube-api-access-27qdl" (OuterVolumeSpecName: "kube-api-access-27qdl") pod "ca225e1d-b41a-4c8c-9121-53becbc73fc1" (UID: "ca225e1d-b41a-4c8c-9121-53becbc73fc1"). InnerVolumeSpecName "kube-api-access-27qdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:20:32.902285 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:32.902271    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ca225e1d-b41a-4c8c-9121-53becbc73fc1" (UID: "ca225e1d-b41a-4c8c-9121-53becbc73fc1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:20:33.001575 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.001534    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:20:33.001575 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.001567    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27qdl\" (UniqueName: \"kubernetes.io/projected/ca225e1d-b41a-4c8c-9121-53becbc73fc1-kube-api-access-27qdl\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:20:33.001575 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.001577    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-trusted-ca-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:20:33.001575 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.001586    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-oauth-serving-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:20:33.001835 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.001596    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca225e1d-b41a-4c8c-9121-53becbc73fc1-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:20:33.001835 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.001605    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-serving-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:20:33.001835 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.001613    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca225e1d-b41a-4c8c-9121-53becbc73fc1-console-oauth-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:20:33.657281 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.657250    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65dcbf9b66-rsf9k_ca225e1d-b41a-4c8c-9121-53becbc73fc1/console/0.log"
May 28 03:20:33.657712 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.657292    2569 generic.go:358] "Generic (PLEG): container finished" podID="ca225e1d-b41a-4c8c-9121-53becbc73fc1" containerID="963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7" exitCode=2
May 28 03:20:33.657712 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.657323    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dcbf9b66-rsf9k" event={"ID":"ca225e1d-b41a-4c8c-9121-53becbc73fc1","Type":"ContainerDied","Data":"963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7"}
May 28 03:20:33.657712 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.657380    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dcbf9b66-rsf9k"
May 28 03:20:33.657712 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.657395    2569 scope.go:117] "RemoveContainer" containerID="963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7"
May 28 03:20:33.657712 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.657383    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dcbf9b66-rsf9k" event={"ID":"ca225e1d-b41a-4c8c-9121-53becbc73fc1","Type":"ContainerDied","Data":"42f7cde63aca9c5b263a16cd979d7b490241c4a1539ffdbc9ebbdfaa897a3d9a"}
May 28 03:20:33.666011 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.665994    2569 scope.go:117] "RemoveContainer" containerID="963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7"
May 28 03:20:33.666290 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:20:33.666272    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7\": container with ID starting with 963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7 not found: ID does not exist" containerID="963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7"
May 28 03:20:33.666336 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.666300    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7"} err="failed to get container status \"963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7\": rpc error: code = NotFound desc = could not find container \"963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7\": container with ID starting with 963f7fc66f9422ac2c5a9cf4691d8e73c33182895e5d6c7383efae5c57c53bd7 not found: ID does not exist"
May 28 03:20:33.677866 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.677839    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dcbf9b66-rsf9k"]
May 28 03:20:33.682152 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:33.682128    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65dcbf9b66-rsf9k"]
May 28 03:20:34.865720 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:20:34.865687    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca225e1d-b41a-4c8c-9121-53becbc73fc1" path="/var/lib/kubelet/pods/ca225e1d-b41a-4c8c-9121-53becbc73fc1/volumes"
May 28 03:21:00.340561 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:00.340517    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:00.359724 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:00.359690    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:00.632134 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:00.632040    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:21:00.634351 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:00.634320    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1025fde6-0949-48f6-af56-3ecd15e95ff7-metrics-certs\") pod \"network-metrics-daemon-4z6ww\" (UID: \"1025fde6-0949-48f6-af56-3ecd15e95ff7\") " pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:21:00.755988 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:00.755961    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:00.766167 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:00.766145    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vzb4w\""
May 28 03:21:00.774457 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:00.774440    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4z6ww"
May 28 03:21:00.918792 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:00.918765    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4z6ww"]
May 28 03:21:00.921208 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:21:00.921178    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1025fde6_0949_48f6_af56_3ecd15e95ff7.slice/crio-502a2227b83d69efc9a82d0215d09226fc7916025607cc5f36f4939d2ffbb4a7 WatchSource:0}: Error finding container 502a2227b83d69efc9a82d0215d09226fc7916025607cc5f36f4939d2ffbb4a7: Status 404 returned error can't find the container with id 502a2227b83d69efc9a82d0215d09226fc7916025607cc5f36f4939d2ffbb4a7
May 28 03:21:01.744071 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:01.744041    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4z6ww" event={"ID":"1025fde6-0949-48f6-af56-3ecd15e95ff7","Type":"ContainerStarted","Data":"502a2227b83d69efc9a82d0215d09226fc7916025607cc5f36f4939d2ffbb4a7"}
May 28 03:21:02.748465 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:02.748428    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4z6ww" event={"ID":"1025fde6-0949-48f6-af56-3ecd15e95ff7","Type":"ContainerStarted","Data":"ea05f4701f8b9e979a729106115881e7d662476c3958cf4ea564c43e2640744e"}
May 28 03:21:02.748465 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:02.748464    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4z6ww" event={"ID":"1025fde6-0949-48f6-af56-3ecd15e95ff7","Type":"ContainerStarted","Data":"4bf965b694fe35a7957bc7d1719660afe1b6f3cabaefb95e14fc6da4e929708e"}
May 28 03:21:02.768934 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:02.768887    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4z6ww" podStartSLOduration=252.552862816 podStartE2EDuration="4m13.768870107s" podCreationTimestamp="2026-05-28 03:16:49 +0000 UTC" firstStartedPulling="2026-05-28 03:21:00.923026514 +0000 UTC m=+252.638547188" lastFinishedPulling="2026-05-28 03:21:02.139033806 +0000 UTC m=+253.854554479" observedRunningTime="2026-05-28 03:21:02.76811786 +0000 UTC m=+254.483638552" watchObservedRunningTime="2026-05-28 03:21:02.768870107 +0000 UTC m=+254.484390799"
May 28 03:21:18.385320 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.385235    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 03:21:18.385816 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.385692    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="prometheus" containerID="cri-o://fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38" gracePeriod=600
May 28 03:21:18.385816 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.385730    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy" containerID="cri-o://f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116" gracePeriod=600
May 28 03:21:18.385816 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.385770    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="config-reloader" containerID="cri-o://6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911" gracePeriod=600
May 28 03:21:18.385816 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.385777    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy-thanos" containerID="cri-o://df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b" gracePeriod=600
May 28 03:21:18.386022 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.385749    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="thanos-sidecar" containerID="cri-o://ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8" gracePeriod=600
May 28 03:21:18.386022 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.385795    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy-web" containerID="cri-o://4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8" gracePeriod=600
May 28 03:21:18.644175 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.644150    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:18.787763 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.787718    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-web-config\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.787763 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.787768    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-tls-assets\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.787986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.787805    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-kubelet-serving-ca-bundle\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.787986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.787835    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-trusted-ca-bundle\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.787986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.787891    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-rulefiles-0\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.787986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.787917    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.787986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.787948    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-kube-rbac-proxy\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.787986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.787979    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-metrics-client-certs\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788263 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788009    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfhtr\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-kube-api-access-hfhtr\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788263 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788034    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-db\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788263 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788065    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-config-out\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788263 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788093    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-grpc-tls\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788263 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788118    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-thanos-prometheus-http-client-file\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788263 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788143    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-config\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788263 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788175    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788263 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788217    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-metrics-client-ca\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788263 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788261    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-serving-certs-ca-bundle\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788742 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788292    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-tls\") pod \"02315d19-0379-410e-a14f-89f5a6af1e53\" (UID: \"02315d19-0379-410e-a14f-89f5a6af1e53\") "
May 28 03:21:18.788742 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788321    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:21:18.788742 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788473    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:21:18.788742 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788581    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.788742 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788601    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.788991 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788769    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:21:18.788991 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.788967    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:21:18.790438 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.789631    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 03:21:18.790566 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.790438    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:21:18.791367 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.791246    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:21:18.791367 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.791298    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:21:18.792399 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.792371    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:21:18.792606 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.792572    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:21:18.792677 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.792653    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:21:18.792782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.792754    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:21:18.793445 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.793392    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-config-out" (OuterVolumeSpecName: "config-out") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 03:21:18.793445 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.793395    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-config" (OuterVolumeSpecName: "config") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:21:18.793445 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.793419    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-kube-api-access-hfhtr" (OuterVolumeSpecName: "kube-api-access-hfhtr") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "kube-api-access-hfhtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:21:18.793445 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.793424    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:21:18.793882 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.793859    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:21:18.797721 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797696    2569 generic.go:358] "Generic (PLEG): container finished" podID="02315d19-0379-410e-a14f-89f5a6af1e53" containerID="df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b" exitCode=0
May 28 03:21:18.797815 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797724    2569 generic.go:358] "Generic (PLEG): container finished" podID="02315d19-0379-410e-a14f-89f5a6af1e53" containerID="f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116" exitCode=0
May 28 03:21:18.797815 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797734    2569 generic.go:358] "Generic (PLEG): container finished" podID="02315d19-0379-410e-a14f-89f5a6af1e53" containerID="4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8" exitCode=0
May 28 03:21:18.797815 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797743    2569 generic.go:358] "Generic (PLEG): container finished" podID="02315d19-0379-410e-a14f-89f5a6af1e53" containerID="ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8" exitCode=0
May 28 03:21:18.797815 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797750    2569 generic.go:358] "Generic (PLEG): container finished" podID="02315d19-0379-410e-a14f-89f5a6af1e53" containerID="6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911" exitCode=0
May 28 03:21:18.797815 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797759    2569 generic.go:358] "Generic (PLEG): container finished" podID="02315d19-0379-410e-a14f-89f5a6af1e53" containerID="fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38" exitCode=0
May 28 03:21:18.797815 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797780    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerDied","Data":"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"}
May 28 03:21:18.797815 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797799    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:18.798089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797830    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerDied","Data":"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"}
May 28 03:21:18.798089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797845    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerDied","Data":"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"}
May 28 03:21:18.798089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797854    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerDied","Data":"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"}
May 28 03:21:18.798089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797864    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerDied","Data":"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"}
May 28 03:21:18.798089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797874    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerDied","Data":"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"}
May 28 03:21:18.798089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797885    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02315d19-0379-410e-a14f-89f5a6af1e53","Type":"ContainerDied","Data":"a879065360d6d95d653f632b901d1e891b0eb6599f8be8578ede1e39938381d1"}
May 28 03:21:18.798089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.797897    2569 scope.go:117] "RemoveContainer" containerID="df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"
May 28 03:21:18.805207 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.805183    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-web-config" (OuterVolumeSpecName: "web-config") pod "02315d19-0379-410e-a14f-89f5a6af1e53" (UID: "02315d19-0379-410e-a14f-89f5a6af1e53"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:21:18.806829 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.806811    2569 scope.go:117] "RemoveContainer" containerID="f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"
May 28 03:21:18.815296 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.815276    2569 scope.go:117] "RemoveContainer" containerID="4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"
May 28 03:21:18.822851 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.822832    2569 scope.go:117] "RemoveContainer" containerID="ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"
May 28 03:21:18.829397 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.829379    2569 scope.go:117] "RemoveContainer" containerID="6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"
May 28 03:21:18.835476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.835459    2569 scope.go:117] "RemoveContainer" containerID="fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"
May 28 03:21:18.842144 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.842125    2569 scope.go:117] "RemoveContainer" containerID="11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"
May 28 03:21:18.849167 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.849138    2569 scope.go:117] "RemoveContainer" containerID="df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"
May 28 03:21:18.849420 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:21:18.849386    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": container with ID starting with df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b not found: ID does not exist" containerID="df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"
May 28 03:21:18.849511 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.849429    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"} err="failed to get container status \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": rpc error: code = NotFound desc = could not find container \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": container with ID starting with df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b not found: ID does not exist"
May 28 03:21:18.849511 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.849456    2569 scope.go:117] "RemoveContainer" containerID="f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"
May 28 03:21:18.849678 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:21:18.849656    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": container with ID starting with f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116 not found: ID does not exist" containerID="f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"
May 28 03:21:18.849723 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.849688    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"} err="failed to get container status \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": rpc error: code = NotFound desc = could not find container \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": container with ID starting with f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116 not found: ID does not exist"
May 28 03:21:18.849723 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.849711    2569 scope.go:117] "RemoveContainer" containerID="4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"
May 28 03:21:18.849908 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:21:18.849888    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": container with ID starting with 4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8 not found: ID does not exist" containerID="4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"
May 28 03:21:18.849970 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.849914    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"} err="failed to get container status \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": rpc error: code = NotFound desc = could not find container \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": container with ID starting with 4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8 not found: ID does not exist"
May 28 03:21:18.849970 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.849933    2569 scope.go:117] "RemoveContainer" containerID="ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"
May 28 03:21:18.850186 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:21:18.850157    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": container with ID starting with ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8 not found: ID does not exist" containerID="ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"
May 28 03:21:18.850233 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.850190    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"} err="failed to get container status \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": rpc error: code = NotFound desc = could not find container \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": container with ID starting with ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8 not found: ID does not exist"
May 28 03:21:18.850233 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.850207    2569 scope.go:117] "RemoveContainer" containerID="6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"
May 28 03:21:18.850472 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:21:18.850442    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": container with ID starting with 6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911 not found: ID does not exist" containerID="6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"
May 28 03:21:18.850576 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.850469    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"} err="failed to get container status \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": rpc error: code = NotFound desc = could not find container \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": container with ID starting with 6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911 not found: ID does not exist"
May 28 03:21:18.850576 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.850490    2569 scope.go:117] "RemoveContainer" containerID="fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"
May 28 03:21:18.850714 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:21:18.850697    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": container with ID starting with fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38 not found: ID does not exist" containerID="fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"
May 28 03:21:18.850767 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.850718    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"} err="failed to get container status \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": rpc error: code = NotFound desc = could not find container \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": container with ID starting with fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38 not found: ID does not exist"
May 28 03:21:18.850767 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.850734    2569 scope.go:117] "RemoveContainer" containerID="11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"
May 28 03:21:18.850941 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:21:18.850916    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": container with ID starting with 11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438 not found: ID does not exist" containerID="11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"
May 28 03:21:18.851031 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.850945    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"} err="failed to get container status \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": rpc error: code = NotFound desc = could not find container \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": container with ID starting with 11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438 not found: ID does not exist"
May 28 03:21:18.851031 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.850965    2569 scope.go:117] "RemoveContainer" containerID="df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"
May 28 03:21:18.851225 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.851202    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"} err="failed to get container status \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": rpc error: code = NotFound desc = could not find container \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": container with ID starting with df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b not found: ID does not exist"
May 28 03:21:18.851225 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.851225    2569 scope.go:117] "RemoveContainer" containerID="f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"
May 28 03:21:18.851492 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.851471    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"} err="failed to get container status \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": rpc error: code = NotFound desc = could not find container \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": container with ID starting with f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116 not found: ID does not exist"
May 28 03:21:18.851563 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.851494    2569 scope.go:117] "RemoveContainer" containerID="4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"
May 28 03:21:18.851705 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.851673    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"} err="failed to get container status \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": rpc error: code = NotFound desc = could not find container \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": container with ID starting with 4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8 not found: ID does not exist"
May 28 03:21:18.851767 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.851705    2569 scope.go:117] "RemoveContainer" containerID="ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"
May 28 03:21:18.851882 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.851861    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"} err="failed to get container status \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": rpc error: code = NotFound desc = could not find container \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": container with ID starting with ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8 not found: ID does not exist"
May 28 03:21:18.851930 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.851883    2569 scope.go:117] "RemoveContainer" containerID="6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"
May 28 03:21:18.852103 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852080    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"} err="failed to get container status \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": rpc error: code = NotFound desc = could not find container \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": container with ID starting with 6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911 not found: ID does not exist"
May 28 03:21:18.852152 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852107    2569 scope.go:117] "RemoveContainer" containerID="fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"
May 28 03:21:18.852303 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852284    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"} err="failed to get container status \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": rpc error: code = NotFound desc = could not find container \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": container with ID starting with fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38 not found: ID does not exist"
May 28 03:21:18.852393 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852305    2569 scope.go:117] "RemoveContainer" containerID="11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"
May 28 03:21:18.852561 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852545    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"} err="failed to get container status \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": rpc error: code = NotFound desc = could not find container \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": container with ID starting with 11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438 not found: ID does not exist"
May 28 03:21:18.852606 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852562    2569 scope.go:117] "RemoveContainer" containerID="df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"
May 28 03:21:18.852758 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852739    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"} err="failed to get container status \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": rpc error: code = NotFound desc = could not find container \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": container with ID starting with df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b not found: ID does not exist"
May 28 03:21:18.852819 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852761    2569 scope.go:117] "RemoveContainer" containerID="f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"
May 28 03:21:18.852952 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852932    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"} err="failed to get container status \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": rpc error: code = NotFound desc = could not find container \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": container with ID starting with f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116 not found: ID does not exist"
May 28 03:21:18.852989 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.852953    2569 scope.go:117] "RemoveContainer" containerID="4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"
May 28 03:21:18.853152 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.853132    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"} err="failed to get container status \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": rpc error: code = NotFound desc = could not find container \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": container with ID starting with 4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8 not found: ID does not exist"
May 28 03:21:18.853205 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.853153    2569 scope.go:117] "RemoveContainer" containerID="ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"
May 28 03:21:18.853375 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.853333    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"} err="failed to get container status \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": rpc error: code = NotFound desc = could not find container \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": container with ID starting with ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8 not found: ID does not exist"
May 28 03:21:18.853434 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.853377    2569 scope.go:117] "RemoveContainer" containerID="6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"
May 28 03:21:18.853614 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.853597    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"} err="failed to get container status \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": rpc error: code = NotFound desc = could not find container \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": container with ID starting with 6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911 not found: ID does not exist"
May 28 03:21:18.853671 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.853617    2569 scope.go:117] "RemoveContainer" containerID="fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"
May 28 03:21:18.853832 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.853810    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"} err="failed to get container status \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": rpc error: code = NotFound desc = could not find container \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": container with ID starting with fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38 not found: ID does not exist"
May 28 03:21:18.853882 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.853835    2569 scope.go:117] "RemoveContainer" containerID="11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"
May 28 03:21:18.854048 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854025    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"} err="failed to get container status \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": rpc error: code = NotFound desc = could not find container \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": container with ID starting with 11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438 not found: ID does not exist"
May 28 03:21:18.854112 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854049    2569 scope.go:117] "RemoveContainer" containerID="df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"
May 28 03:21:18.854266 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854241    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"} err="failed to get container status \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": rpc error: code = NotFound desc = could not find container \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": container with ID starting with df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b not found: ID does not exist"
May 28 03:21:18.854266 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854265    2569 scope.go:117] "RemoveContainer" containerID="f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"
May 28 03:21:18.854463 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854445    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"} err="failed to get container status \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": rpc error: code = NotFound desc = could not find container \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": container with ID starting with f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116 not found: ID does not exist"
May 28 03:21:18.854516 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854465    2569 scope.go:117] "RemoveContainer" containerID="4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"
May 28 03:21:18.854665 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854640    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"} err="failed to get container status \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": rpc error: code = NotFound desc = could not find container \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": container with ID starting with 4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8 not found: ID does not exist"
May 28 03:21:18.854665 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854664    2569 scope.go:117] "RemoveContainer" containerID="ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"
May 28 03:21:18.854855 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854837    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"} err="failed to get container status \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": rpc error: code = NotFound desc = could not find container \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": container with ID starting with ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8 not found: ID does not exist"
May 28 03:21:18.854911 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.854856    2569 scope.go:117] "RemoveContainer" containerID="6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"
May 28 03:21:18.855058 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855037    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"} err="failed to get container status \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": rpc error: code = NotFound desc = could not find container \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": container with ID starting with 6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911 not found: ID does not exist"
May 28 03:21:18.855119 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855059    2569 scope.go:117] "RemoveContainer" containerID="fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"
May 28 03:21:18.855280 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855261    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"} err="failed to get container status \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": rpc error: code = NotFound desc = could not find container \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": container with ID starting with fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38 not found: ID does not exist"
May 28 03:21:18.855330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855280    2569 scope.go:117] "RemoveContainer" containerID="11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"
May 28 03:21:18.855501 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855483    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"} err="failed to get container status \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": rpc error: code = NotFound desc = could not find container \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": container with ID starting with 11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438 not found: ID does not exist"
May 28 03:21:18.855544 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855502    2569 scope.go:117] "RemoveContainer" containerID="df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"
May 28 03:21:18.855708 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855691    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"} err="failed to get container status \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": rpc error: code = NotFound desc = could not find container \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": container with ID starting with df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b not found: ID does not exist"
May 28 03:21:18.855748 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855709    2569 scope.go:117] "RemoveContainer" containerID="f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"
May 28 03:21:18.855886 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855870    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"} err="failed to get container status \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": rpc error: code = NotFound desc = could not find container \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": container with ID starting with f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116 not found: ID does not exist"
May 28 03:21:18.855928 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.855886    2569 scope.go:117] "RemoveContainer" containerID="4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"
May 28 03:21:18.856053 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856038    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"} err="failed to get container status \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": rpc error: code = NotFound desc = could not find container \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": container with ID starting with 4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8 not found: ID does not exist"
May 28 03:21:18.856100 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856054    2569 scope.go:117] "RemoveContainer" containerID="ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"
May 28 03:21:18.856257 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856240    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"} err="failed to get container status \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": rpc error: code = NotFound desc = could not find container \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": container with ID starting with ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8 not found: ID does not exist"
May 28 03:21:18.856257 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856257    2569 scope.go:117] "RemoveContainer" containerID="6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"
May 28 03:21:18.856453 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856436    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"} err="failed to get container status \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": rpc error: code = NotFound desc = could not find container \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": container with ID starting with 6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911 not found: ID does not exist"
May 28 03:21:18.856492 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856453    2569 scope.go:117] "RemoveContainer" containerID="fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"
May 28 03:21:18.856644 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856628    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"} err="failed to get container status \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": rpc error: code = NotFound desc = could not find container \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": container with ID starting with fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38 not found: ID does not exist"
May 28 03:21:18.856698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856643    2569 scope.go:117] "RemoveContainer" containerID="11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"
May 28 03:21:18.856810 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856794    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"} err="failed to get container status \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": rpc error: code = NotFound desc = could not find container \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": container with ID starting with 11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438 not found: ID does not exist"
May 28 03:21:18.856848 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856810    2569 scope.go:117] "RemoveContainer" containerID="df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"
May 28 03:21:18.856974 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856957    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b"} err="failed to get container status \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": rpc error: code = NotFound desc = could not find container \"df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b\": container with ID starting with df0e54f040c84a89c472a79dd6b530085c18657098352a2554f51e9cd271ff5b not found: ID does not exist"
May 28 03:21:18.857016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.856974    2569 scope.go:117] "RemoveContainer" containerID="f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"
May 28 03:21:18.857171 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857142    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116"} err="failed to get container status \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": rpc error: code = NotFound desc = could not find container \"f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116\": container with ID starting with f75f6cd3aa167dbfb6443a9b96fa120fff6867edb7003d90be9171a0dc650116 not found: ID does not exist"
May 28 03:21:18.857218 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857172    2569 scope.go:117] "RemoveContainer" containerID="4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"
May 28 03:21:18.857374 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857357    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8"} err="failed to get container status \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": rpc error: code = NotFound desc = could not find container \"4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8\": container with ID starting with 4b5ed5b03f7bf41680cdc59a20b0267a9e0994921cf44f38acf734914fd3bfb8 not found: ID does not exist"
May 28 03:21:18.857414 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857375    2569 scope.go:117] "RemoveContainer" containerID="ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"
May 28 03:21:18.857605 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857583    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8"} err="failed to get container status \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": rpc error: code = NotFound desc = could not find container \"ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8\": container with ID starting with ef1dbe6872c70992c281796557345b06c1c8f0427e960bec59b067edf5a671e8 not found: ID does not exist"
May 28 03:21:18.857643 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857606    2569 scope.go:117] "RemoveContainer" containerID="6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"
May 28 03:21:18.857810 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857791    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911"} err="failed to get container status \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": rpc error: code = NotFound desc = could not find container \"6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911\": container with ID starting with 6d41683c7614833ceaac0a6cba66aa366430e9cc5b521c5d96206b151e1d0911 not found: ID does not exist"
May 28 03:21:18.857860 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857811    2569 scope.go:117] "RemoveContainer" containerID="fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"
May 28 03:21:18.857985 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857970    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38"} err="failed to get container status \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": rpc error: code = NotFound desc = could not find container \"fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38\": container with ID starting with fb34965155817412bd833ca7b6c95315960cc5fcc9ed90de52783cd4fe53ac38 not found: ID does not exist"
May 28 03:21:18.858034 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.857985    2569 scope.go:117] "RemoveContainer" containerID="11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"
May 28 03:21:18.858181 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.858164    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438"} err="failed to get container status \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": rpc error: code = NotFound desc = could not find container \"11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438\": container with ID starting with 11073137a0380ba9ac56004e31ef28816217f76efd1618ad6b226029d0a75438 not found: ID does not exist"
May 28 03:21:18.889259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889194    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889220    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889232    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-kube-rbac-proxy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889243    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-metrics-client-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889254    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhtr\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-kube-api-access-hfhtr\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889259 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889264    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-prometheus-k8s-db\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889273    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02315d19-0379-410e-a14f-89f5a6af1e53-config-out\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889282    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-grpc-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889291    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889299    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889308    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889316    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-metrics-client-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889325    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02315d19-0379-410e-a14f-89f5a6af1e53-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889334    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889364    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02315d19-0379-410e-a14f-89f5a6af1e53-web-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:18.889518 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:18.889376    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02315d19-0379-410e-a14f-89f5a6af1e53-tls-assets\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:21:19.116220 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.116188    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 03:21:19.124750 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.124719    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 03:21:19.151127 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151047    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 03:21:19.151369 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151326    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca225e1d-b41a-4c8c-9121-53becbc73fc1" containerName="console"
May 28 03:21:19.151369 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151356    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca225e1d-b41a-4c8c-9121-53becbc73fc1" containerName="console"
May 28 03:21:19.151369 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151366    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="init-config-reloader"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151373    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="init-config-reloader"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151384    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy-thanos"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151389    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy-thanos"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151397    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="prometheus"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151405    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="prometheus"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151418    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151424    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151434    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="thanos-sidecar"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151439    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="thanos-sidecar"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151444    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy-web"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151449    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy-web"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151456    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="config-reloader"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151460    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="config-reloader"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151508    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="thanos-sidecar"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151517    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca225e1d-b41a-4c8c-9121-53becbc73fc1" containerName="console"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151526    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy-thanos"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151533    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy-web"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151539    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="kube-rbac-proxy"
May 28 03:21:19.151535 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151546    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="prometheus"
May 28 03:21:19.152251 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.151552    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" containerName="config-reloader"
May 28 03:21:19.155470 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.155452    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.158156 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158134    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 28 03:21:19.158265 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158135    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 28 03:21:19.158265 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158178    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 28 03:21:19.158686 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158447    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n2f9h\""
May 28 03:21:19.158686 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158484    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7b9v9a1fougje\""
May 28 03:21:19.158686 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158521    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 28 03:21:19.158686 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158542    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 28 03:21:19.158686 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158484    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 28 03:21:19.158686 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158486    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 28 03:21:19.158686 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158488    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 28 03:21:19.158686 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.158682    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 28 03:21:19.159075 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.159028    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 28 03:21:19.160441 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.160427    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 28 03:21:19.167732 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.167703    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 28 03:21:19.171129 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.171106    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 28 03:21:19.172069 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.172049    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 03:21:19.293267 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293235    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c9904fd4-14d7-4c08-a9c3-54a38676aec0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293267 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293489 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293489 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293317    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293489 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293336    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9904fd4-14d7-4c08-a9c3-54a38676aec0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293489 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-config\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293489 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293489 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293468    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293489 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293502    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9904fd4-14d7-4c08-a9c3-54a38676aec0-config-out\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293601    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293616    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-web-config\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293647    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fpdj\" (UniqueName: \"kubernetes.io/projected/c9904fd4-14d7-4c08-a9c3-54a38676aec0-kube-api-access-5fpdj\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.293739 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.293680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.394245 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.394207    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.394690 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.394259    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.394690 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.394282    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9904fd4-14d7-4c08-a9c3-54a38676aec0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.394690 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.394302    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-config\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.394690 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.394324    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.394690 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.394366    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.394690 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.394384    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.394690 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.394402    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395135 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395104    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395267 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395157    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395267 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395166    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9904fd4-14d7-4c08-a9c3-54a38676aec0-config-out\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395267 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395224    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395267 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395259    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395515 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395293    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395515 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395317    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-web-config\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395515 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395364    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5fpdj\" (UniqueName: \"kubernetes.io/projected/c9904fd4-14d7-4c08-a9c3-54a38676aec0-kube-api-access-5fpdj\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395515 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395417    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395515 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395460    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c9904fd4-14d7-4c08-a9c3-54a38676aec0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395515 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395490    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.395817 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.395579    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.396410 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.396037    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.397208 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.397038    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c9904fd4-14d7-4c08-a9c3-54a38676aec0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.397883 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.397601    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.397883 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.397705    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.398331 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.398308    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.398458 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.398365    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-config\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.398758 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.398736    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.399242 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.399218    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.399993 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.399965    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.400089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.400008    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.400262 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.400244    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9904fd4-14d7-4c08-a9c3-54a38676aec0-config-out\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.400394 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.400379    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-web-config\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.400496 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.400474    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9904fd4-14d7-4c08-a9c3-54a38676aec0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.400791 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.400774    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c9904fd4-14d7-4c08-a9c3-54a38676aec0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.400869 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.400854    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9904fd4-14d7-4c08-a9c3-54a38676aec0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.405801 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.405755    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fpdj\" (UniqueName: \"kubernetes.io/projected/c9904fd4-14d7-4c08-a9c3-54a38676aec0-kube-api-access-5fpdj\") pod \"prometheus-k8s-0\" (UID: \"c9904fd4-14d7-4c08-a9c3-54a38676aec0\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.465954 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.465922    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:19.618227 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.618197    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 03:21:19.621247 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:21:19.621213    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9904fd4_14d7_4c08_a9c3_54a38676aec0.slice/crio-9eb55fd3df754e3fb5d30fd33165d41d89eac0c0e35a59c12a77a254b67746de WatchSource:0}: Error finding container 9eb55fd3df754e3fb5d30fd33165d41d89eac0c0e35a59c12a77a254b67746de: Status 404 returned error can't find the container with id 9eb55fd3df754e3fb5d30fd33165d41d89eac0c0e35a59c12a77a254b67746de
May 28 03:21:19.802334 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.802297    2569 generic.go:358] "Generic (PLEG): container finished" podID="c9904fd4-14d7-4c08-a9c3-54a38676aec0" containerID="808e7fcae35b1030537d077c531046837a5fba10bea2625c147fe8f8b177758c" exitCode=0
May 28 03:21:19.802516 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.802391    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9904fd4-14d7-4c08-a9c3-54a38676aec0","Type":"ContainerDied","Data":"808e7fcae35b1030537d077c531046837a5fba10bea2625c147fe8f8b177758c"}
May 28 03:21:19.802516 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:19.802436    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9904fd4-14d7-4c08-a9c3-54a38676aec0","Type":"ContainerStarted","Data":"9eb55fd3df754e3fb5d30fd33165d41d89eac0c0e35a59c12a77a254b67746de"}
May 28 03:21:20.809113 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:20.809064    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9904fd4-14d7-4c08-a9c3-54a38676aec0","Type":"ContainerStarted","Data":"5da5d106990346a7c20d6e5b2c96a2f017a2482331b8b20dc6e29bd252c61659"}
May 28 03:21:20.809113 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:20.809117    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9904fd4-14d7-4c08-a9c3-54a38676aec0","Type":"ContainerStarted","Data":"6ca0a2e88912b2d6cc3cbc564e66cdef5b26ce10903574828e08264c0dbaa2dd"}
May 28 03:21:20.809546 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:20.809133    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9904fd4-14d7-4c08-a9c3-54a38676aec0","Type":"ContainerStarted","Data":"e3697a3d2ce36e6ea87553e5242134a661aa6a97ab2a6870aee6aa663723771f"}
May 28 03:21:20.809546 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:20.809145    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9904fd4-14d7-4c08-a9c3-54a38676aec0","Type":"ContainerStarted","Data":"de41b6854de2d98c9e6ad5ea402aaa8c8d4eb60c48c7c7e79da2f32e2a6b13ad"}
May 28 03:21:20.809546 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:20.809157    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9904fd4-14d7-4c08-a9c3-54a38676aec0","Type":"ContainerStarted","Data":"01794470fff484920549d84790aef712e825e540d52610ba3dedcd6328aca886"}
May 28 03:21:20.809546 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:20.809168    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9904fd4-14d7-4c08-a9c3-54a38676aec0","Type":"ContainerStarted","Data":"d9c026fbe6cdee679b059e2388674c212f8eaac1891f7d89db6b5221927492c6"}
May 28 03:21:20.851314 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:20.851262    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.851244613 podStartE2EDuration="1.851244613s" podCreationTimestamp="2026-05-28 03:21:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 03:21:20.848980358 +0000 UTC m=+272.564501050" watchObservedRunningTime="2026-05-28 03:21:20.851244613 +0000 UTC m=+272.566765306"
May 28 03:21:20.865802 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:20.865774    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02315d19-0379-410e-a14f-89f5a6af1e53" path="/var/lib/kubelet/pods/02315d19-0379-410e-a14f-89f5a6af1e53/volumes"
May 28 03:21:24.466330 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:24.466284    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:21:48.761639 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:48.761478    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:21:48.761954 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:48.761819    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:21:48.771154 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:21:48.771133    2569 kubelet.go:1628] "Image garbage collection succeeded"
May 28 03:22:19.466381 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:22:19.466320    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:22:19.481909 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:22:19.481882    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:22:20.002479 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:22:20.002452    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 28 03:24:52.222788 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.222749    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"]
May 28 03:24:52.225880 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.225857    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.228876 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.228847    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l2nrl\""
May 28 03:24:52.229306 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.229286    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
May 28 03:24:52.229463 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.229335    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
May 28 03:24:52.229463 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.229393    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
May 28 03:24:52.232599 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.232578    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
May 28 03:24:52.250418 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.250388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e41963f3-1896-49f7-bed3-d2c3cb1f4ee3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77cd789dcd-jfj78\" (UID: \"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3\") " pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.250540 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.250423    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6k7z\" (UniqueName: \"kubernetes.io/projected/e41963f3-1896-49f7-bed3-d2c3cb1f4ee3-kube-api-access-c6k7z\") pod \"opendatahub-operator-controller-manager-77cd789dcd-jfj78\" (UID: \"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3\") " pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.250540 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.250450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e41963f3-1896-49f7-bed3-d2c3cb1f4ee3-webhook-cert\") pod \"opendatahub-operator-controller-manager-77cd789dcd-jfj78\" (UID: \"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3\") " pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.266782 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.266752    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"]
May 28 03:24:52.351059 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.351016    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e41963f3-1896-49f7-bed3-d2c3cb1f4ee3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77cd789dcd-jfj78\" (UID: \"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3\") " pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.351059 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.351062    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c6k7z\" (UniqueName: \"kubernetes.io/projected/e41963f3-1896-49f7-bed3-d2c3cb1f4ee3-kube-api-access-c6k7z\") pod \"opendatahub-operator-controller-manager-77cd789dcd-jfj78\" (UID: \"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3\") " pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.351312 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.351093    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e41963f3-1896-49f7-bed3-d2c3cb1f4ee3-webhook-cert\") pod \"opendatahub-operator-controller-manager-77cd789dcd-jfj78\" (UID: \"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3\") " pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.353523 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.353499    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e41963f3-1896-49f7-bed3-d2c3cb1f4ee3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77cd789dcd-jfj78\" (UID: \"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3\") " pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.353642 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.353560    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e41963f3-1896-49f7-bed3-d2c3cb1f4ee3-webhook-cert\") pod \"opendatahub-operator-controller-manager-77cd789dcd-jfj78\" (UID: \"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3\") " pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.363610 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.363584    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6k7z\" (UniqueName: \"kubernetes.io/projected/e41963f3-1896-49f7-bed3-d2c3cb1f4ee3-kube-api-access-c6k7z\") pod \"opendatahub-operator-controller-manager-77cd789dcd-jfj78\" (UID: \"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3\") " pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.536982 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.536894    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:52.669215 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.669159    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"]
May 28 03:24:52.672197 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:24:52.672165    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41963f3_1896_49f7_bed3_d2c3cb1f4ee3.slice/crio-2e196265c278176099995890cac991a472731850b1f16badcb0f40dd76b26530 WatchSource:0}: Error finding container 2e196265c278176099995890cac991a472731850b1f16badcb0f40dd76b26530: Status 404 returned error can't find the container with id 2e196265c278176099995890cac991a472731850b1f16badcb0f40dd76b26530
May 28 03:24:52.673814 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:52.673798    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 03:24:53.405820 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:53.405767    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78" event={"ID":"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3","Type":"ContainerStarted","Data":"2e196265c278176099995890cac991a472731850b1f16badcb0f40dd76b26530"}
May 28 03:24:55.414904 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:55.414813    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78" event={"ID":"e41963f3-1896-49f7-bed3-d2c3cb1f4ee3","Type":"ContainerStarted","Data":"f81e38707cf56e29b22d4e3cbc8ac0228526aa2612bcc669c0e5b48a6dc91ea0"}
May 28 03:24:55.414904 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:55.414895    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:24:55.454058 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:24:55.454003    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78" podStartSLOduration=1.032496962 podStartE2EDuration="3.453988879s" podCreationTimestamp="2026-05-28 03:24:52 +0000 UTC" firstStartedPulling="2026-05-28 03:24:52.673915905 +0000 UTC m=+484.389436575" lastFinishedPulling="2026-05-28 03:24:55.095407823 +0000 UTC m=+486.810928492" observedRunningTime="2026-05-28 03:24:55.451537086 +0000 UTC m=+487.167057780" watchObservedRunningTime="2026-05-28 03:24:55.453988879 +0000 UTC m=+487.169509570"
May 28 03:25:06.200619 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.200581    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"]
May 28 03:25:06.204166 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.204146    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.207482 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.207289    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 28 03:25:06.207482 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.207326    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 28 03:25:06.207482 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.207329    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jvk9g\""
May 28 03:25:06.207482 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.207370    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 28 03:25:06.207482 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.207363    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 28 03:25:06.207482 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.207466    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 28 03:25:06.216024 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.215997    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"]
May 28 03:25:06.268429 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.268388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bq5\" (UniqueName: \"kubernetes.io/projected/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-kube-api-access-n4bq5\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.268612 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.268491    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-manager-config\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.268612 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.268539    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-cert\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.268612 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.268555    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-metrics-cert\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.369777 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.369737    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bq5\" (UniqueName: \"kubernetes.io/projected/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-kube-api-access-n4bq5\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.369940 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.369860    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-manager-config\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.369986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.369937    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-cert\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.369986 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.369963    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-metrics-cert\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.370583 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.370556    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-manager-config\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.372575 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.372550    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-cert\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.372671 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.372611    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-metrics-cert\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.382940 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.382903    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bq5\" (UniqueName: \"kubernetes.io/projected/bb4d1987-2d52-4d3b-b6dd-5b1511bed63b-kube-api-access-n4bq5\") pod \"lws-controller-manager-8546f974ff-rx48j\" (UID: \"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b\") " pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.421497 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.421466    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77cd789dcd-jfj78"
May 28 03:25:06.513128 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.513036    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:06.649283 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:06.649173    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"]
May 28 03:25:06.652413 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:25:06.652385    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4d1987_2d52_4d3b_b6dd_5b1511bed63b.slice/crio-50b2a0026f6058ea3b21c03950247abdffda7d69d18896757a3865e1431311a2 WatchSource:0}: Error finding container 50b2a0026f6058ea3b21c03950247abdffda7d69d18896757a3865e1431311a2: Status 404 returned error can't find the container with id 50b2a0026f6058ea3b21c03950247abdffda7d69d18896757a3865e1431311a2
May 28 03:25:07.454832 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:07.454797    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j" event={"ID":"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b","Type":"ContainerStarted","Data":"50b2a0026f6058ea3b21c03950247abdffda7d69d18896757a3865e1431311a2"}
May 28 03:25:10.169534 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.169503    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"]
May 28 03:25:10.172165 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.172143    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.175821 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.175797    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2d89b\""
May 28 03:25:10.175821 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.175811    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
May 28 03:25:10.176001 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.175820    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 28 03:25:10.176001 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.175863    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 28 03:25:10.176001 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.175881    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
May 28 03:25:10.187745 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.187716    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"]
May 28 03:25:10.307220 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.307177    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f-tls-certs\") pod \"kube-auth-proxy-5b78c488db-w5drg\" (UID: \"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f\") " pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.307418 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.307228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qblhj\" (UniqueName: \"kubernetes.io/projected/42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f-kube-api-access-qblhj\") pod \"kube-auth-proxy-5b78c488db-w5drg\" (UID: \"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f\") " pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.307418 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.307358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f-tmp\") pod \"kube-auth-proxy-5b78c488db-w5drg\" (UID: \"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f\") " pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.407749 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.407715    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f-tls-certs\") pod \"kube-auth-proxy-5b78c488db-w5drg\" (UID: \"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f\") " pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.407876 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.407752    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qblhj\" (UniqueName: \"kubernetes.io/projected/42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f-kube-api-access-qblhj\") pod \"kube-auth-proxy-5b78c488db-w5drg\" (UID: \"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f\") " pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.407876 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.407806    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f-tmp\") pod \"kube-auth-proxy-5b78c488db-w5drg\" (UID: \"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f\") " pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.410137 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.410104    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f-tmp\") pod \"kube-auth-proxy-5b78c488db-w5drg\" (UID: \"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f\") " pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.410325 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.410310    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f-tls-certs\") pod \"kube-auth-proxy-5b78c488db-w5drg\" (UID: \"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f\") " pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.423821 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.423800    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qblhj\" (UniqueName: \"kubernetes.io/projected/42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f-kube-api-access-qblhj\") pod \"kube-auth-proxy-5b78c488db-w5drg\" (UID: \"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f\") " pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.485010 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.484980    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"
May 28 03:25:10.615717 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:10.615578    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b78c488db-w5drg"]
May 28 03:25:10.618232 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:25:10.618201    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b7ec75_10d2_43b5_92a0_0d7d1ab6ca4f.slice/crio-46156dead72b323a5e4e1c0892d453dd4e49ec5b02fc4a6bc17b75bc124adb08 WatchSource:0}: Error finding container 46156dead72b323a5e4e1c0892d453dd4e49ec5b02fc4a6bc17b75bc124adb08: Status 404 returned error can't find the container with id 46156dead72b323a5e4e1c0892d453dd4e49ec5b02fc4a6bc17b75bc124adb08
May 28 03:25:11.469682 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:11.469640    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j" event={"ID":"bb4d1987-2d52-4d3b-b6dd-5b1511bed63b","Type":"ContainerStarted","Data":"300e9786371a61bd38543e8eee88a83da6e288198216b1ccc29b4dbb4909e79c"}
May 28 03:25:11.470159 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:11.469885    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:25:11.470868 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:11.470832    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg" event={"ID":"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f","Type":"ContainerStarted","Data":"46156dead72b323a5e4e1c0892d453dd4e49ec5b02fc4a6bc17b75bc124adb08"}
May 28 03:25:11.489332 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:11.489281    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j" podStartSLOduration=1.763322263 podStartE2EDuration="5.489264991s" podCreationTimestamp="2026-05-28 03:25:06 +0000 UTC" firstStartedPulling="2026-05-28 03:25:06.654096209 +0000 UTC m=+498.369616880" lastFinishedPulling="2026-05-28 03:25:10.380038927 +0000 UTC m=+502.095559608" observedRunningTime="2026-05-28 03:25:11.486591334 +0000 UTC m=+503.202112026" watchObservedRunningTime="2026-05-28 03:25:11.489264991 +0000 UTC m=+503.204785682"
May 28 03:25:14.483243 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:14.483209    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg" event={"ID":"42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f","Type":"ContainerStarted","Data":"3e94aaa70fca64ca75e72381ae5dd2824bb105536bb6d03c2e7ccc98593fc3ff"}
May 28 03:25:14.501111 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:14.501066    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b78c488db-w5drg" podStartSLOduration=1.18721355 podStartE2EDuration="4.501051387s" podCreationTimestamp="2026-05-28 03:25:10 +0000 UTC" firstStartedPulling="2026-05-28 03:25:10.620120177 +0000 UTC m=+502.335640850" lastFinishedPulling="2026-05-28 03:25:13.933958008 +0000 UTC m=+505.649478687" observedRunningTime="2026-05-28 03:25:14.499643984 +0000 UTC m=+506.215164677" watchObservedRunningTime="2026-05-28 03:25:14.501051387 +0000 UTC m=+506.216572132"
May 28 03:25:22.477687 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:25:22.477645    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8546f974ff-rx48j"
May 28 03:26:48.784537 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:48.784510    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:26:48.786676 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:48.786654    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:26:52.788850 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.788814    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"]
May 28 03:26:52.790958 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.790943    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:52.793682 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.793652    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
May 28 03:26:52.793811 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.793681    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 28 03:26:52.794430 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.794411    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lgq78\""
May 28 03:26:52.794493 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.794413    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 28 03:26:52.794493 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.794417    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
May 28 03:26:52.802363 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.802323    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"]
May 28 03:26:52.878298 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.878255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa58f5a0-4c35-48b5-b636-af3e85a31125-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:52.878487 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.878371    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa58f5a0-4c35-48b5-b636-af3e85a31125-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:52.878487 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.878457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nwql\" (UniqueName: \"kubernetes.io/projected/fa58f5a0-4c35-48b5-b636-af3e85a31125-kube-api-access-5nwql\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:52.979913 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.979857    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa58f5a0-4c35-48b5-b636-af3e85a31125-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:52.980128 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.979952    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5nwql\" (UniqueName: \"kubernetes.io/projected/fa58f5a0-4c35-48b5-b636-af3e85a31125-kube-api-access-5nwql\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:52.980128 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.980020    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa58f5a0-4c35-48b5-b636-af3e85a31125-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:52.980128 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:26:52.980018    2569 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
May 28 03:26:52.980400 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:26:52.980177    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa58f5a0-4c35-48b5-b636-af3e85a31125-plugin-serving-cert podName:fa58f5a0-4c35-48b5-b636-af3e85a31125 nodeName:}" failed. No retries permitted until 2026-05-28 03:26:53.480153773 +0000 UTC m=+605.195674460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/fa58f5a0-4c35-48b5-b636-af3e85a31125-plugin-serving-cert") pod "kuadrant-console-plugin-6cb54b5c86-wcv74" (UID: "fa58f5a0-4c35-48b5-b636-af3e85a31125") : secret "plugin-serving-cert" not found
May 28 03:26:52.980793 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.980770    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa58f5a0-4c35-48b5-b636-af3e85a31125-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:52.991048 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:52.991020    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nwql\" (UniqueName: \"kubernetes.io/projected/fa58f5a0-4c35-48b5-b636-af3e85a31125-kube-api-access-5nwql\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:53.485046 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:53.485008    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa58f5a0-4c35-48b5-b636-af3e85a31125-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:53.487543 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:53.487521    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa58f5a0-4c35-48b5-b636-af3e85a31125-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-wcv74\" (UID: \"fa58f5a0-4c35-48b5-b636-af3e85a31125\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:53.715539 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:53.715498    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"
May 28 03:26:53.855800 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:53.855743    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74"]
May 28 03:26:53.878295 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:26:53.878254    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa58f5a0_4c35_48b5_b636_af3e85a31125.slice/crio-8c4dfeec9d79d6ab61e32e9e8336ced35a02473fa4829baae4d57cf97aa3124f WatchSource:0}: Error finding container 8c4dfeec9d79d6ab61e32e9e8336ced35a02473fa4829baae4d57cf97aa3124f: Status 404 returned error can't find the container with id 8c4dfeec9d79d6ab61e32e9e8336ced35a02473fa4829baae4d57cf97aa3124f
May 28 03:26:54.806908 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:26:54.806865    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74" event={"ID":"fa58f5a0-4c35-48b5-b636-af3e85a31125","Type":"ContainerStarted","Data":"8c4dfeec9d79d6ab61e32e9e8336ced35a02473fa4829baae4d57cf97aa3124f"}
May 28 03:27:18.902961 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:18.902923    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74" event={"ID":"fa58f5a0-4c35-48b5-b636-af3e85a31125","Type":"ContainerStarted","Data":"6d24eeb55fd98be17b457101e9a9befec8b011290bf00fc7ae444f2c5cf49173"}
May 28 03:27:18.920456 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:18.920396    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wcv74" podStartSLOduration=2.696934563 podStartE2EDuration="26.920380077s" podCreationTimestamp="2026-05-28 03:26:52 +0000 UTC" firstStartedPulling="2026-05-28 03:26:53.87952204 +0000 UTC m=+605.595042710" lastFinishedPulling="2026-05-28 03:27:18.102967551 +0000 UTC m=+629.818488224" observedRunningTime="2026-05-28 03:27:18.91768031 +0000 UTC m=+630.633201003" watchObservedRunningTime="2026-05-28 03:27:18.920380077 +0000 UTC m=+630.635900770"
May 28 03:27:36.326786 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.326706    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-8qmwx"]
May 28 03:27:36.334840 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.334819    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:36.340232 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.340200    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-8qmwx"]
May 28 03:27:36.341017 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.340997    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 28 03:27:36.376283 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.376241    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4af45256-0aa6-4812-a81d-d2e8699e1761-config-file\") pod \"limitador-limitador-7d549b5b-8qmwx\" (UID: \"4af45256-0aa6-4812-a81d-d2e8699e1761\") " pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:36.376477 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.376372    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhv7\" (UniqueName: \"kubernetes.io/projected/4af45256-0aa6-4812-a81d-d2e8699e1761-kube-api-access-vhhv7\") pod \"limitador-limitador-7d549b5b-8qmwx\" (UID: \"4af45256-0aa6-4812-a81d-d2e8699e1761\") " pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:36.415975 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.415938    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-8qmwx"]
May 28 03:27:36.477727 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.477682    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4af45256-0aa6-4812-a81d-d2e8699e1761-config-file\") pod \"limitador-limitador-7d549b5b-8qmwx\" (UID: \"4af45256-0aa6-4812-a81d-d2e8699e1761\") " pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:36.477873 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.477779    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhv7\" (UniqueName: \"kubernetes.io/projected/4af45256-0aa6-4812-a81d-d2e8699e1761-kube-api-access-vhhv7\") pod \"limitador-limitador-7d549b5b-8qmwx\" (UID: \"4af45256-0aa6-4812-a81d-d2e8699e1761\") " pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:36.478312 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.478293    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4af45256-0aa6-4812-a81d-d2e8699e1761-config-file\") pod \"limitador-limitador-7d549b5b-8qmwx\" (UID: \"4af45256-0aa6-4812-a81d-d2e8699e1761\") " pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:36.486469 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.486441    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhv7\" (UniqueName: \"kubernetes.io/projected/4af45256-0aa6-4812-a81d-d2e8699e1761-kube-api-access-vhhv7\") pod \"limitador-limitador-7d549b5b-8qmwx\" (UID: \"4af45256-0aa6-4812-a81d-d2e8699e1761\") " pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:36.645191 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.645143    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:36.780105 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.780067    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-8qmwx"]
May 28 03:27:36.783234 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:27:36.783203    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af45256_0aa6_4812_a81d_d2e8699e1761.slice/crio-dda2233da0cf3c385854a8128657550ee365b9933131ef1c13dfe69d44a85da6 WatchSource:0}: Error finding container dda2233da0cf3c385854a8128657550ee365b9933131ef1c13dfe69d44a85da6: Status 404 returned error can't find the container with id dda2233da0cf3c385854a8128657550ee365b9933131ef1c13dfe69d44a85da6
May 28 03:27:36.963595 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:36.963508    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx" event={"ID":"4af45256-0aa6-4812-a81d-d2e8699e1761","Type":"ContainerStarted","Data":"dda2233da0cf3c385854a8128657550ee365b9933131ef1c13dfe69d44a85da6"}
May 28 03:27:37.320875 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.320806    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7498df8756-7kx9v"]
May 28 03:27:37.323910 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.323890    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-7kx9v"
May 28 03:27:37.326331 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.326314    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xlm95\""
May 28 03:27:37.332363 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.332327    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498df8756-7kx9v"]
May 28 03:27:37.385708 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.385663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r77x\" (UniqueName: \"kubernetes.io/projected/1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27-kube-api-access-9r77x\") pod \"authorino-7498df8756-7kx9v\" (UID: \"1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27\") " pod="kuadrant-system/authorino-7498df8756-7kx9v"
May 28 03:27:37.486873 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.486832    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9r77x\" (UniqueName: \"kubernetes.io/projected/1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27-kube-api-access-9r77x\") pod \"authorino-7498df8756-7kx9v\" (UID: \"1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27\") " pod="kuadrant-system/authorino-7498df8756-7kx9v"
May 28 03:27:37.507655 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.507622    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r77x\" (UniqueName: \"kubernetes.io/projected/1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27-kube-api-access-9r77x\") pod \"authorino-7498df8756-7kx9v\" (UID: \"1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27\") " pod="kuadrant-system/authorino-7498df8756-7kx9v"
May 28 03:27:37.634033 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.633997    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-7kx9v"
May 28 03:27:37.809874 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.809838    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498df8756-7kx9v"]
May 28 03:27:37.813195 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:27:37.813157    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b02c584_3ad5_4e30_b01f_e1ddb1f4bf27.slice/crio-fd343359bea0bb82612a55ad7ee0aa34c2caedf0287b5e9a507f9b79bf63fda4 WatchSource:0}: Error finding container fd343359bea0bb82612a55ad7ee0aa34c2caedf0287b5e9a507f9b79bf63fda4: Status 404 returned error can't find the container with id fd343359bea0bb82612a55ad7ee0aa34c2caedf0287b5e9a507f9b79bf63fda4
May 28 03:27:37.968756 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:37.968667    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-7kx9v" event={"ID":"1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27","Type":"ContainerStarted","Data":"fd343359bea0bb82612a55ad7ee0aa34c2caedf0287b5e9a507f9b79bf63fda4"}
May 28 03:27:40.981382 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:40.981319    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx" event={"ID":"4af45256-0aa6-4812-a81d-d2e8699e1761","Type":"ContainerStarted","Data":"5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba"}
May 28 03:27:40.981867 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:40.981600    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:40.999903 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:40.999842    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx" podStartSLOduration=1.5991921279999999 podStartE2EDuration="4.999824534s" podCreationTimestamp="2026-05-28 03:27:36 +0000 UTC" firstStartedPulling="2026-05-28 03:27:36.785286179 +0000 UTC m=+648.500806849" lastFinishedPulling="2026-05-28 03:27:40.185918576 +0000 UTC m=+651.901439255" observedRunningTime="2026-05-28 03:27:40.997806383 +0000 UTC m=+652.713327078" watchObservedRunningTime="2026-05-28 03:27:40.999824534 +0000 UTC m=+652.715345226"
May 28 03:27:43.992089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:43.992051    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-7kx9v" event={"ID":"1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27","Type":"ContainerStarted","Data":"9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1"}
May 28 03:27:44.009297 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:44.009243    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-7498df8756-7kx9v" podStartSLOduration=1.651857146 podStartE2EDuration="7.009227184s" podCreationTimestamp="2026-05-28 03:27:37 +0000 UTC" firstStartedPulling="2026-05-28 03:27:37.814875863 +0000 UTC m=+649.530396537" lastFinishedPulling="2026-05-28 03:27:43.172245902 +0000 UTC m=+654.887766575" observedRunningTime="2026-05-28 03:27:44.007289654 +0000 UTC m=+655.722810356" watchObservedRunningTime="2026-05-28 03:27:44.009227184 +0000 UTC m=+655.724747876"
May 28 03:27:51.041113 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.041082    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-8qmwx"]
May 28 03:27:51.041644 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.041319    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx" podUID="4af45256-0aa6-4812-a81d-d2e8699e1761" containerName="limitador" containerID="cri-o://5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba" gracePeriod=30
May 28 03:27:51.042032 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.042013    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:51.646546 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.646520    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:51.718437 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.718400    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhhv7\" (UniqueName: \"kubernetes.io/projected/4af45256-0aa6-4812-a81d-d2e8699e1761-kube-api-access-vhhv7\") pod \"4af45256-0aa6-4812-a81d-d2e8699e1761\" (UID: \"4af45256-0aa6-4812-a81d-d2e8699e1761\") "
May 28 03:27:51.718629 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.718458    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4af45256-0aa6-4812-a81d-d2e8699e1761-config-file\") pod \"4af45256-0aa6-4812-a81d-d2e8699e1761\" (UID: \"4af45256-0aa6-4812-a81d-d2e8699e1761\") "
May 28 03:27:51.718824 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.718799    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af45256-0aa6-4812-a81d-d2e8699e1761-config-file" (OuterVolumeSpecName: "config-file") pod "4af45256-0aa6-4812-a81d-d2e8699e1761" (UID: "4af45256-0aa6-4812-a81d-d2e8699e1761"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 03:27:51.720683 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.720657    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af45256-0aa6-4812-a81d-d2e8699e1761-kube-api-access-vhhv7" (OuterVolumeSpecName: "kube-api-access-vhhv7") pod "4af45256-0aa6-4812-a81d-d2e8699e1761" (UID: "4af45256-0aa6-4812-a81d-d2e8699e1761"). InnerVolumeSpecName "kube-api-access-vhhv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:27:51.819411 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.819379    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhv7\" (UniqueName: \"kubernetes.io/projected/4af45256-0aa6-4812-a81d-d2e8699e1761-kube-api-access-vhhv7\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:27:51.819411 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:51.819408    2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4af45256-0aa6-4812-a81d-d2e8699e1761-config-file\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:27:52.021452 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.021362    2569 generic.go:358] "Generic (PLEG): container finished" podID="4af45256-0aa6-4812-a81d-d2e8699e1761" containerID="5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba" exitCode=0
May 28 03:27:52.021452 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.021424    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx"
May 28 03:27:52.021646 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.021449    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx" event={"ID":"4af45256-0aa6-4812-a81d-d2e8699e1761","Type":"ContainerDied","Data":"5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba"}
May 28 03:27:52.021646 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.021488    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-8qmwx" event={"ID":"4af45256-0aa6-4812-a81d-d2e8699e1761","Type":"ContainerDied","Data":"dda2233da0cf3c385854a8128657550ee365b9933131ef1c13dfe69d44a85da6"}
May 28 03:27:52.021646 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.021510    2569 scope.go:117] "RemoveContainer" containerID="5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba"
May 28 03:27:52.029726 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.029706    2569 scope.go:117] "RemoveContainer" containerID="5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba"
May 28 03:27:52.029997 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:27:52.029976    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba\": container with ID starting with 5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba not found: ID does not exist" containerID="5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba"
May 28 03:27:52.030080 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.030005    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba"} err="failed to get container status \"5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba\": rpc error: code = NotFound desc = could not find container \"5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba\": container with ID starting with 5d967b3adfb11de7e4f016dce2bf86d551e5e21e6f02387298f9f033e13576ba not found: ID does not exist"
May 28 03:27:52.044619 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.044584    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-8qmwx"]
May 28 03:27:52.046876 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.046855    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-8qmwx"]
May 28 03:27:52.374408 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.374378    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-68c4fbbd6f-26php"]
May 28 03:27:52.375012 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.374780    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af45256-0aa6-4812-a81d-d2e8699e1761" containerName="limitador"
May 28 03:27:52.375012 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.374797    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af45256-0aa6-4812-a81d-d2e8699e1761" containerName="limitador"
May 28 03:27:52.375188 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.375062    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="4af45256-0aa6-4812-a81d-d2e8699e1761" containerName="limitador"
May 28 03:27:52.379755 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.379735    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:27:52.382985 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.382969    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
May 28 03:27:52.383073 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.382975    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9bq8c\""
May 28 03:27:52.393000 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.392976    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-68c4fbbd6f-26php"]
May 28 03:27:52.525017 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.524981    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxvx\" (UniqueName: \"kubernetes.io/projected/597d824b-3198-478f-bd10-0e93574496ba-kube-api-access-txxvx\") pod \"postgres-68c4fbbd6f-26php\" (UID: \"597d824b-3198-478f-bd10-0e93574496ba\") " pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:27:52.525210 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.525039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/597d824b-3198-478f-bd10-0e93574496ba-data\") pod \"postgres-68c4fbbd6f-26php\" (UID: \"597d824b-3198-478f-bd10-0e93574496ba\") " pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:27:52.625644 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.625558    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-txxvx\" (UniqueName: \"kubernetes.io/projected/597d824b-3198-478f-bd10-0e93574496ba-kube-api-access-txxvx\") pod \"postgres-68c4fbbd6f-26php\" (UID: \"597d824b-3198-478f-bd10-0e93574496ba\") " pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:27:52.625644 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.625615    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/597d824b-3198-478f-bd10-0e93574496ba-data\") pod \"postgres-68c4fbbd6f-26php\" (UID: \"597d824b-3198-478f-bd10-0e93574496ba\") " pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:27:52.626027 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.626008    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/597d824b-3198-478f-bd10-0e93574496ba-data\") pod \"postgres-68c4fbbd6f-26php\" (UID: \"597d824b-3198-478f-bd10-0e93574496ba\") " pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:27:52.635811 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.635782    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxvx\" (UniqueName: \"kubernetes.io/projected/597d824b-3198-478f-bd10-0e93574496ba-kube-api-access-txxvx\") pod \"postgres-68c4fbbd6f-26php\" (UID: \"597d824b-3198-478f-bd10-0e93574496ba\") " pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:27:52.690790 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.690736    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:27:52.816652 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.816577    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-68c4fbbd6f-26php"]
May 28 03:27:52.818782 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:27:52.818756    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597d824b_3198_478f_bd10_0e93574496ba.slice/crio-275b098d7e6c57dafcaf1bf077b1e5c313e0c289063e1c8a7ab8bfe3ed244282 WatchSource:0}: Error finding container 275b098d7e6c57dafcaf1bf077b1e5c313e0c289063e1c8a7ab8bfe3ed244282: Status 404 returned error can't find the container with id 275b098d7e6c57dafcaf1bf077b1e5c313e0c289063e1c8a7ab8bfe3ed244282
May 28 03:27:52.866161 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:52.866130    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af45256-0aa6-4812-a81d-d2e8699e1761" path="/var/lib/kubelet/pods/4af45256-0aa6-4812-a81d-d2e8699e1761/volumes"
May 28 03:27:53.026429 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:27:53.026321    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-68c4fbbd6f-26php" event={"ID":"597d824b-3198-478f-bd10-0e93574496ba","Type":"ContainerStarted","Data":"275b098d7e6c57dafcaf1bf077b1e5c313e0c289063e1c8a7ab8bfe3ed244282"}
May 28 03:28:02.058615 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:02.058576    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-68c4fbbd6f-26php" event={"ID":"597d824b-3198-478f-bd10-0e93574496ba","Type":"ContainerStarted","Data":"d50b86aa3cd7fad2161ee458409a3f5dc2b4f650ef850887be8d6b1d3e7f4caf"}
May 28 03:28:02.059007 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:02.058704    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:28:02.077398 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:02.077330    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/postgres-68c4fbbd6f-26php" podStartSLOduration=1.765325177 podStartE2EDuration="10.077314374s" podCreationTimestamp="2026-05-28 03:27:52 +0000 UTC" firstStartedPulling="2026-05-28 03:27:52.820070025 +0000 UTC m=+664.535590695" lastFinishedPulling="2026-05-28 03:28:01.132059219 +0000 UTC m=+672.847579892" observedRunningTime="2026-05-28 03:28:02.075869572 +0000 UTC m=+673.791390278" watchObservedRunningTime="2026-05-28 03:28:02.077314374 +0000 UTC m=+673.792835067"
May 28 03:28:08.089904 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:08.089867    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-68c4fbbd6f-26php"
May 28 03:28:08.879832 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:08.879800    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6df8c55f54-d6hjm"]
May 28 03:28:08.883023 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:08.883005    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6df8c55f54-d6hjm"
May 28 03:28:08.891004 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:08.890972    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6df8c55f54-d6hjm"]
May 28 03:28:08.967989 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:08.967951    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5ll4\" (UniqueName: \"kubernetes.io/projected/276a651d-ccd5-40bd-bba4-1fd57073cf3a-kube-api-access-n5ll4\") pod \"authorino-6df8c55f54-d6hjm\" (UID: \"276a651d-ccd5-40bd-bba4-1fd57073cf3a\") " pod="kuadrant-system/authorino-6df8c55f54-d6hjm"
May 28 03:28:09.050192 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.050160    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6df8c55f54-d6hjm"]
May 28 03:28:09.050439 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:28:09.050420    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-n5ll4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6df8c55f54-d6hjm" podUID="276a651d-ccd5-40bd-bba4-1fd57073cf3a"
May 28 03:28:09.068797 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.068761    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n5ll4\" (UniqueName: \"kubernetes.io/projected/276a651d-ccd5-40bd-bba4-1fd57073cf3a-kube-api-access-n5ll4\") pod \"authorino-6df8c55f54-d6hjm\" (UID: \"276a651d-ccd5-40bd-bba4-1fd57073cf3a\") " pod="kuadrant-system/authorino-6df8c55f54-d6hjm"
May 28 03:28:09.080439 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.080405    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6df8c55f54-d6hjm"
May 28 03:28:09.082199 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.082179    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f8457c85d-gp7hc"]
May 28 03:28:09.083599 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.083581    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5ll4\" (UniqueName: \"kubernetes.io/projected/276a651d-ccd5-40bd-bba4-1fd57073cf3a-kube-api-access-n5ll4\") pod \"authorino-6df8c55f54-d6hjm\" (UID: \"276a651d-ccd5-40bd-bba4-1fd57073cf3a\") " pod="kuadrant-system/authorino-6df8c55f54-d6hjm"
May 28 03:28:09.084702 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.084688    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:28:09.086968 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.086938    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
May 28 03:28:09.092471 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.092451    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6df8c55f54-d6hjm"
May 28 03:28:09.097061 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.097036    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f8457c85d-gp7hc"]
May 28 03:28:09.169916 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.169824    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5ll4\" (UniqueName: \"kubernetes.io/projected/276a651d-ccd5-40bd-bba4-1fd57073cf3a-kube-api-access-n5ll4\") pod \"276a651d-ccd5-40bd-bba4-1fd57073cf3a\" (UID: \"276a651d-ccd5-40bd-bba4-1fd57073cf3a\") "
May 28 03:28:09.170072 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.169997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f347da-59cb-4d27-9394-b6b84ee8364b-tls-cert\") pod \"authorino-5f8457c85d-gp7hc\" (UID: \"00f347da-59cb-4d27-9394-b6b84ee8364b\") " pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:28:09.170122 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.170082    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8vf\" (UniqueName: \"kubernetes.io/projected/00f347da-59cb-4d27-9394-b6b84ee8364b-kube-api-access-vd8vf\") pod \"authorino-5f8457c85d-gp7hc\" (UID: \"00f347da-59cb-4d27-9394-b6b84ee8364b\") " pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:28:09.171934 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.171908    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276a651d-ccd5-40bd-bba4-1fd57073cf3a-kube-api-access-n5ll4" (OuterVolumeSpecName: "kube-api-access-n5ll4") pod "276a651d-ccd5-40bd-bba4-1fd57073cf3a" (UID: "276a651d-ccd5-40bd-bba4-1fd57073cf3a"). InnerVolumeSpecName "kube-api-access-n5ll4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:28:09.270661 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.270626    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8vf\" (UniqueName: \"kubernetes.io/projected/00f347da-59cb-4d27-9394-b6b84ee8364b-kube-api-access-vd8vf\") pod \"authorino-5f8457c85d-gp7hc\" (UID: \"00f347da-59cb-4d27-9394-b6b84ee8364b\") " pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:28:09.270831 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.270691    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f347da-59cb-4d27-9394-b6b84ee8364b-tls-cert\") pod \"authorino-5f8457c85d-gp7hc\" (UID: \"00f347da-59cb-4d27-9394-b6b84ee8364b\") " pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:28:09.270831 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.270731    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5ll4\" (UniqueName: \"kubernetes.io/projected/276a651d-ccd5-40bd-bba4-1fd57073cf3a-kube-api-access-n5ll4\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:28:09.273026 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.273000    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f347da-59cb-4d27-9394-b6b84ee8364b-tls-cert\") pod \"authorino-5f8457c85d-gp7hc\" (UID: \"00f347da-59cb-4d27-9394-b6b84ee8364b\") " pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:28:09.278577 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.278556    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8vf\" (UniqueName: \"kubernetes.io/projected/00f347da-59cb-4d27-9394-b6b84ee8364b-kube-api-access-vd8vf\") pod \"authorino-5f8457c85d-gp7hc\" (UID: \"00f347da-59cb-4d27-9394-b6b84ee8364b\") " pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:28:09.401053 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.401013    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:28:09.525628 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:09.525599    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f8457c85d-gp7hc"]
May 28 03:28:09.527971 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:28:09.527947    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00f347da_59cb_4d27_9394_b6b84ee8364b.slice/crio-c453fa2497e6f3fc3e7e556f5094bd2fa1f869ca7904a96b26a0ed76a4b1fa69 WatchSource:0}: Error finding container c453fa2497e6f3fc3e7e556f5094bd2fa1f869ca7904a96b26a0ed76a4b1fa69: Status 404 returned error can't find the container with id c453fa2497e6f3fc3e7e556f5094bd2fa1f869ca7904a96b26a0ed76a4b1fa69
May 28 03:28:10.084897 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.084810    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f8457c85d-gp7hc" event={"ID":"00f347da-59cb-4d27-9394-b6b84ee8364b","Type":"ContainerStarted","Data":"0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb"}
May 28 03:28:10.084897 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.084858    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f8457c85d-gp7hc" event={"ID":"00f347da-59cb-4d27-9394-b6b84ee8364b","Type":"ContainerStarted","Data":"c453fa2497e6f3fc3e7e556f5094bd2fa1f869ca7904a96b26a0ed76a4b1fa69"}
May 28 03:28:10.084897 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.084895    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6df8c55f54-d6hjm"
May 28 03:28:10.101114 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.101075    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-5f8457c85d-gp7hc" podStartSLOduration=0.82534649 podStartE2EDuration="1.101061109s" podCreationTimestamp="2026-05-28 03:28:09 +0000 UTC" firstStartedPulling="2026-05-28 03:28:09.529259784 +0000 UTC m=+681.244780456" lastFinishedPulling="2026-05-28 03:28:09.804974401 +0000 UTC m=+681.520495075" observedRunningTime="2026-05-28 03:28:10.100717797 +0000 UTC m=+681.816238492" watchObservedRunningTime="2026-05-28 03:28:10.101061109 +0000 UTC m=+681.816581807"
May 28 03:28:10.126220 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.126171    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6df8c55f54-d6hjm"]
May 28 03:28:10.134123 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.134092    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6df8c55f54-d6hjm"]
May 28 03:28:10.135567 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.135544    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498df8756-7kx9v"]
May 28 03:28:10.135833 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.135811    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-7498df8756-7kx9v" podUID="1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27" containerName="authorino" containerID="cri-o://9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1" gracePeriod=30
May 28 03:28:10.377916 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.377889    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-7kx9v"
May 28 03:28:10.480064 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.480032    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r77x\" (UniqueName: \"kubernetes.io/projected/1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27-kube-api-access-9r77x\") pod \"1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27\" (UID: \"1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27\") "
May 28 03:28:10.482100 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.482075    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27-kube-api-access-9r77x" (OuterVolumeSpecName: "kube-api-access-9r77x") pod "1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27" (UID: "1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27"). InnerVolumeSpecName "kube-api-access-9r77x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:28:10.580603 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.580566    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r77x\" (UniqueName: \"kubernetes.io/projected/1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27-kube-api-access-9r77x\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:28:10.866220 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:10.866175    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276a651d-ccd5-40bd-bba4-1fd57073cf3a" path="/var/lib/kubelet/pods/276a651d-ccd5-40bd-bba4-1fd57073cf3a/volumes"
May 28 03:28:11.088705 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:11.088671    2569 generic.go:358] "Generic (PLEG): container finished" podID="1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27" containerID="9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1" exitCode=0
May 28 03:28:11.088883 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:11.088732    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-7kx9v"
May 28 03:28:11.088883 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:11.088759    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-7kx9v" event={"ID":"1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27","Type":"ContainerDied","Data":"9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1"}
May 28 03:28:11.088883 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:11.088807    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-7kx9v" event={"ID":"1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27","Type":"ContainerDied","Data":"fd343359bea0bb82612a55ad7ee0aa34c2caedf0287b5e9a507f9b79bf63fda4"}
May 28 03:28:11.088883 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:11.088824    2569 scope.go:117] "RemoveContainer" containerID="9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1"
May 28 03:28:11.096709 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:11.096691    2569 scope.go:117] "RemoveContainer" containerID="9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1"
May 28 03:28:11.096963 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:28:11.096943    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1\": container with ID starting with 9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1 not found: ID does not exist" containerID="9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1"
May 28 03:28:11.097032 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:11.096975    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1"} err="failed to get container status \"9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1\": rpc error: code = NotFound desc = could not find container \"9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1\": container with ID starting with 9290ed2f1fccab74c0e8e74a2f81df0108b00399d1a4b05f5c6d8ceb12f002e1 not found: ID does not exist"
May 28 03:28:11.104523 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:11.104495    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498df8756-7kx9v"]
May 28 03:28:11.108411 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:11.108390    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7498df8756-7kx9v"]
May 28 03:28:12.866052 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:28:12.866019    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27" path="/var/lib/kubelet/pods/1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27/volumes"
May 28 03:29:22.095313 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.095277    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"]
May 28 03:29:22.095791 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.095627    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27" containerName="authorino"
May 28 03:29:22.095791 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.095639    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27" containerName="authorino"
May 28 03:29:22.095791 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.095706    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="1b02c584-3ad5-4e30-b01f-e1ddb1f4bf27" containerName="authorino"
May 28 03:29:22.098925 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.098907    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.102041 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.102019    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
May 28 03:29:22.102041 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.102037    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
May 28 03:29:22.102213 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.102043    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9vdxf\""
May 28 03:29:22.102213 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.102020    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
May 28 03:29:22.106817 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.106795    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"]
May 28 03:29:22.187125 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.187089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-model-cache\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.187279 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.187152    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6caf3d-b90f-4953-8dc5-46aca15a4139-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.187279 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.187248    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-kserve-provision-location\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.187367 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.187291    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbclv\" (UniqueName: \"kubernetes.io/projected/fc6caf3d-b90f-4953-8dc5-46aca15a4139-kube-api-access-zbclv\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.187411 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.187361    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-dshm\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.187490 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.187428    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-home\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.287989 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.287952    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6caf3d-b90f-4953-8dc5-46aca15a4139-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.287989 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.287993    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-kserve-provision-location\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.288241 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.288013    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zbclv\" (UniqueName: \"kubernetes.io/projected/fc6caf3d-b90f-4953-8dc5-46aca15a4139-kube-api-access-zbclv\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.288241 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.288031    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-dshm\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.288241 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.288063    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-home\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.288241 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.288120    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-model-cache\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.288493 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.288468    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-kserve-provision-location\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.288537 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.288487    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-home\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.288576 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.288566    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-model-cache\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.290376 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.290332    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6caf3d-b90f-4953-8dc5-46aca15a4139-dshm\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.290650 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.290632    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6caf3d-b90f-4953-8dc5-46aca15a4139-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.296691 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.296667    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbclv\" (UniqueName: \"kubernetes.io/projected/fc6caf3d-b90f-4953-8dc5-46aca15a4139-kube-api-access-zbclv\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6\" (UID: \"fc6caf3d-b90f-4953-8dc5-46aca15a4139\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.409982 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.409888    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:22.535323 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:22.535299    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"]
May 28 03:29:22.537301 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:29:22.537273    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6caf3d_b90f_4953_8dc5_46aca15a4139.slice/crio-620b38f8d17f74c860e2af619ef8a87ef6f056b8de28109808112a2b7fe1b19f WatchSource:0}: Error finding container 620b38f8d17f74c860e2af619ef8a87ef6f056b8de28109808112a2b7fe1b19f: Status 404 returned error can't find the container with id 620b38f8d17f74c860e2af619ef8a87ef6f056b8de28109808112a2b7fe1b19f
May 28 03:29:23.095046 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.095013    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"]
May 28 03:29:23.098732 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.098707    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.101145 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.101124    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
May 28 03:29:23.108420 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.108040    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"]
May 28 03:29:23.196576 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.196533    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-kserve-provision-location\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.196775 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.196595    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-home\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.196775 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.196674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.196775 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.196697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zkc\" (UniqueName: \"kubernetes.io/projected/22c825cb-f657-4146-8af1-1e3505fdf3f4-kube-api-access-74zkc\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.196775 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.196721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-dshm\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.196775 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.196740    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c825cb-f657-4146-8af1-1e3505fdf3f4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.297984 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.297951    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-kserve-provision-location\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.298167 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.298007    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-home\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.298167 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.298085    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.298167 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.298111    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74zkc\" (UniqueName: \"kubernetes.io/projected/22c825cb-f657-4146-8af1-1e3505fdf3f4-kube-api-access-74zkc\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.298167 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.298137    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-dshm\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.298167 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.298159    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c825cb-f657-4146-8af1-1e3505fdf3f4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.298476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.298438    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-kserve-provision-location\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.298538 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.298510    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.298592 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.298508    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-home\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.300823 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.300782    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c825cb-f657-4146-8af1-1e3505fdf3f4-dshm\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.301204 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.301175    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c825cb-f657-4146-8af1-1e3505fdf3f4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.307130 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.307106    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zkc\" (UniqueName: \"kubernetes.io/projected/22c825cb-f657-4146-8af1-1e3505fdf3f4-kube-api-access-74zkc\") pod \"e2e-trlp-test-simulated-kserve-84db68679b-52brt\" (UID: \"22c825cb-f657-4146-8af1-1e3505fdf3f4\") " pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.325606 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.325566    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6" event={"ID":"fc6caf3d-b90f-4953-8dc5-46aca15a4139","Type":"ContainerStarted","Data":"620b38f8d17f74c860e2af619ef8a87ef6f056b8de28109808112a2b7fe1b19f"}
May 28 03:29:23.413483 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.413398    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:23.563527 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:23.563498    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"]
May 28 03:29:23.566434 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:29:23.566401    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c825cb_f657_4146_8af1_1e3505fdf3f4.slice/crio-957aa390b55fbfc100ed65f01a6941f471447dd620132f4e589548f2f52b9f70 WatchSource:0}: Error finding container 957aa390b55fbfc100ed65f01a6941f471447dd620132f4e589548f2f52b9f70: Status 404 returned error can't find the container with id 957aa390b55fbfc100ed65f01a6941f471447dd620132f4e589548f2f52b9f70
May 28 03:29:24.331212 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:24.331146    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt" event={"ID":"22c825cb-f657-4146-8af1-1e3505fdf3f4","Type":"ContainerStarted","Data":"957aa390b55fbfc100ed65f01a6941f471447dd620132f4e589548f2f52b9f70"}
May 28 03:29:28.346128 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:28.346083    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt" event={"ID":"22c825cb-f657-4146-8af1-1e3505fdf3f4","Type":"ContainerStarted","Data":"878e9e019c201a0ee6909f41affc3bb70659009a864857a44a80f03859df4959"}
May 28 03:29:28.355057 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:28.355012    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6" event={"ID":"fc6caf3d-b90f-4953-8dc5-46aca15a4139","Type":"ContainerStarted","Data":"cc19637d3c981a0df9e9925bfffb4a7a1e75bace23fbfd16a5129f7fd7b30dd2"}
May 28 03:29:34.376756 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:34.376719    2569 generic.go:358] "Generic (PLEG): container finished" podID="fc6caf3d-b90f-4953-8dc5-46aca15a4139" containerID="cc19637d3c981a0df9e9925bfffb4a7a1e75bace23fbfd16a5129f7fd7b30dd2" exitCode=0
May 28 03:29:34.377216 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:34.376793    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6" event={"ID":"fc6caf3d-b90f-4953-8dc5-46aca15a4139","Type":"ContainerDied","Data":"cc19637d3c981a0df9e9925bfffb4a7a1e75bace23fbfd16a5129f7fd7b30dd2"}
May 28 03:29:34.378260 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:34.378241    2569 generic.go:358] "Generic (PLEG): container finished" podID="22c825cb-f657-4146-8af1-1e3505fdf3f4" containerID="878e9e019c201a0ee6909f41affc3bb70659009a864857a44a80f03859df4959" exitCode=0
May 28 03:29:34.378334 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:34.378287    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt" event={"ID":"22c825cb-f657-4146-8af1-1e3505fdf3f4","Type":"ContainerDied","Data":"878e9e019c201a0ee6909f41affc3bb70659009a864857a44a80f03859df4959"}
May 28 03:29:36.387047 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:36.387010    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt" event={"ID":"22c825cb-f657-4146-8af1-1e3505fdf3f4","Type":"ContainerStarted","Data":"70a36089810214989d8a58f1d2320e647e0ddec2281949eb1c8b15c7bd9298cc"}
May 28 03:29:36.387502 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:36.387242    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:36.388521 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:36.388490    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6" event={"ID":"fc6caf3d-b90f-4953-8dc5-46aca15a4139","Type":"ContainerStarted","Data":"d71525735f45248ef106529fea258d9b22974220c655169896aa190a4513181f"}
May 28 03:29:36.388712 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:36.388696    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:29:36.408054 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:36.407997    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt" podStartSLOduration=1.468274761 podStartE2EDuration="13.407982966s" podCreationTimestamp="2026-05-28 03:29:23 +0000 UTC" firstStartedPulling="2026-05-28 03:29:23.568802616 +0000 UTC m=+755.284323299" lastFinishedPulling="2026-05-28 03:29:35.508510834 +0000 UTC m=+767.224031504" observedRunningTime="2026-05-28 03:29:36.406015442 +0000 UTC m=+768.121536136" watchObservedRunningTime="2026-05-28 03:29:36.407982966 +0000 UTC m=+768.123503658"
May 28 03:29:36.424049 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:36.423977    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6" podStartSLOduration=1.458594335 podStartE2EDuration="14.423964621s" podCreationTimestamp="2026-05-28 03:29:22 +0000 UTC" firstStartedPulling="2026-05-28 03:29:22.538962957 +0000 UTC m=+754.254483644" lastFinishedPulling="2026-05-28 03:29:35.504333253 +0000 UTC m=+767.219853930" observedRunningTime="2026-05-28 03:29:36.423084082 +0000 UTC m=+768.138604774" watchObservedRunningTime="2026-05-28 03:29:36.423964621 +0000 UTC m=+768.139485406"
May 28 03:29:47.411912 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:47.411878    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-84db68679b-52brt"
May 28 03:29:47.412463 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:29:47.412364    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6"
May 28 03:30:25.798463 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.798373    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"]
May 28 03:30:25.802235 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.802217    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.804628 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.804605    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
May 28 03:30:25.811369 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.811327    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"]
May 28 03:30:25.865808 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.865761    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-dshm\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.866016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.865819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-home\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.866016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.865920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.866016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.865967    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.866016 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.866008    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kcxk\" (UniqueName: \"kubernetes.io/projected/f5a91282-477a-4737-9132-c6f1f2cecdb8-kube-api-access-4kcxk\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.866205 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.866033    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a91282-477a-4737-9132-c6f1f2cecdb8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.966945 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.966912    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcxk\" (UniqueName: \"kubernetes.io/projected/f5a91282-477a-4737-9132-c6f1f2cecdb8-kube-api-access-4kcxk\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.967063 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.966959    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a91282-477a-4737-9132-c6f1f2cecdb8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.967063 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.967001    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-dshm\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.967063 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.967020    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-home\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.967063 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.967062    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.967256 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.967103    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.967582 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.967545    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.967683 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.967554    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-home\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.967683 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.967607    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.969414 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.969392    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a91282-477a-4737-9132-c6f1f2cecdb8-dshm\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.969713 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.969695    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a91282-477a-4737-9132-c6f1f2cecdb8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:25.983906 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:25.983868    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kcxk\" (UniqueName: \"kubernetes.io/projected/f5a91282-477a-4737-9132-c6f1f2cecdb8-kube-api-access-4kcxk\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p\" (UID: \"f5a91282-477a-4737-9132-c6f1f2cecdb8\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:26.112443 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:26.112410    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:26.240806 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:26.240778    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"]
May 28 03:30:26.243617 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:30:26.243585    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a91282_477a_4737_9132_c6f1f2cecdb8.slice/crio-795a7b0c33c84a7a64ae1b6308a03938051df9299392649c7f62741faa890583 WatchSource:0}: Error finding container 795a7b0c33c84a7a64ae1b6308a03938051df9299392649c7f62741faa890583: Status 404 returned error can't find the container with id 795a7b0c33c84a7a64ae1b6308a03938051df9299392649c7f62741faa890583
May 28 03:30:26.245244 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:26.245228    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 03:30:26.561520 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:26.561428    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p" event={"ID":"f5a91282-477a-4737-9132-c6f1f2cecdb8","Type":"ContainerStarted","Data":"578d525359d186b4eaa3ba80ab40371d53e65769ef43b909da314b4505dc28f8"}
May 28 03:30:26.561520 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:26.561478    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p" event={"ID":"f5a91282-477a-4737-9132-c6f1f2cecdb8","Type":"ContainerStarted","Data":"795a7b0c33c84a7a64ae1b6308a03938051df9299392649c7f62741faa890583"}
May 28 03:30:32.586074 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:32.586037    2569 generic.go:358] "Generic (PLEG): container finished" podID="f5a91282-477a-4737-9132-c6f1f2cecdb8" containerID="578d525359d186b4eaa3ba80ab40371d53e65769ef43b909da314b4505dc28f8" exitCode=0
May 28 03:30:32.586074 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:32.586079    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p" event={"ID":"f5a91282-477a-4737-9132-c6f1f2cecdb8","Type":"ContainerDied","Data":"578d525359d186b4eaa3ba80ab40371d53e65769ef43b909da314b4505dc28f8"}
May 28 03:30:33.591492 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:33.591454    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p" event={"ID":"f5a91282-477a-4737-9132-c6f1f2cecdb8","Type":"ContainerStarted","Data":"7e42cddff397a15c790f0f2031e7c5f2562510620fbc642a4fac5762bb266bba"}
May 28 03:30:33.591963 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:33.591721    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:33.611031 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:33.610961    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p" podStartSLOduration=8.423358588 podStartE2EDuration="8.610942249s" podCreationTimestamp="2026-05-28 03:30:25 +0000 UTC" firstStartedPulling="2026-05-28 03:30:32.586701512 +0000 UTC m=+824.302222182" lastFinishedPulling="2026-05-28 03:30:32.774285174 +0000 UTC m=+824.489805843" observedRunningTime="2026-05-28 03:30:33.6102248 +0000 UTC m=+825.325745491" watchObservedRunningTime="2026-05-28 03:30:33.610942249 +0000 UTC m=+825.326462954"
May 28 03:30:44.608738 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:44.608705    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p"
May 28 03:30:47.824052 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:47.824013    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f8457c85d-gp7hc"]
May 28 03:30:47.824474 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:47.824217    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-5f8457c85d-gp7hc" podUID="00f347da-59cb-4d27-9394-b6b84ee8364b" containerName="authorino" containerID="cri-o://0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb" gracePeriod=30
May 28 03:30:48.071889 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.071866    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:30:48.170297 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.170264    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f347da-59cb-4d27-9394-b6b84ee8364b-tls-cert\") pod \"00f347da-59cb-4d27-9394-b6b84ee8364b\" (UID: \"00f347da-59cb-4d27-9394-b6b84ee8364b\") "
May 28 03:30:48.170490 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.170313    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd8vf\" (UniqueName: \"kubernetes.io/projected/00f347da-59cb-4d27-9394-b6b84ee8364b-kube-api-access-vd8vf\") pod \"00f347da-59cb-4d27-9394-b6b84ee8364b\" (UID: \"00f347da-59cb-4d27-9394-b6b84ee8364b\") "
May 28 03:30:48.172334 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.172306    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f347da-59cb-4d27-9394-b6b84ee8364b-kube-api-access-vd8vf" (OuterVolumeSpecName: "kube-api-access-vd8vf") pod "00f347da-59cb-4d27-9394-b6b84ee8364b" (UID: "00f347da-59cb-4d27-9394-b6b84ee8364b"). InnerVolumeSpecName "kube-api-access-vd8vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 03:30:48.180180 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.180149    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f347da-59cb-4d27-9394-b6b84ee8364b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "00f347da-59cb-4d27-9394-b6b84ee8364b" (UID: "00f347da-59cb-4d27-9394-b6b84ee8364b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 03:30:48.271493 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.271461    2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f347da-59cb-4d27-9394-b6b84ee8364b-tls-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:30:48.271493 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.271489    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8vf\" (UniqueName: \"kubernetes.io/projected/00f347da-59cb-4d27-9394-b6b84ee8364b-kube-api-access-vd8vf\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
May 28 03:30:48.644703 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.644667    2569 generic.go:358] "Generic (PLEG): container finished" podID="00f347da-59cb-4d27-9394-b6b84ee8364b" containerID="0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb" exitCode=0
May 28 03:30:48.644878 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.644722    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f8457c85d-gp7hc"
May 28 03:30:48.644878 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.644737    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f8457c85d-gp7hc" event={"ID":"00f347da-59cb-4d27-9394-b6b84ee8364b","Type":"ContainerDied","Data":"0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb"}
May 28 03:30:48.644878 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.644771    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f8457c85d-gp7hc" event={"ID":"00f347da-59cb-4d27-9394-b6b84ee8364b","Type":"ContainerDied","Data":"c453fa2497e6f3fc3e7e556f5094bd2fa1f869ca7904a96b26a0ed76a4b1fa69"}
May 28 03:30:48.644878 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.644790    2569 scope.go:117] "RemoveContainer" containerID="0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb"
May 28 03:30:48.653420 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.653394    2569 scope.go:117] "RemoveContainer" containerID="0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb"
May 28 03:30:48.653660 ip-10-0-141-29 kubenswrapper[2569]: E0528 03:30:48.653644    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb\": container with ID starting with 0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb not found: ID does not exist" containerID="0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb"
May 28 03:30:48.653706 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.653666    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb"} err="failed to get container status \"0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb\": rpc error: code = NotFound desc = could not find container \"0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb\": container with ID starting with 0433cbaeb8a8f379820b8712744acba4fb72700337e6c6708a79946ec7a55acb not found: ID does not exist"
May 28 03:30:48.668168 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.668141    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f8457c85d-gp7hc"]
May 28 03:30:48.671439 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.671417    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f8457c85d-gp7hc"]
May 28 03:30:48.870678 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:30:48.870643    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f347da-59cb-4d27-9394-b6b84ee8364b" path="/var/lib/kubelet/pods/00f347da-59cb-4d27-9394-b6b84ee8364b/volumes"
May 28 03:31:48.814510 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:31:48.814480    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:31:48.818161 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:31:48.818138    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:36:48.838542 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:36:48.838512    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:36:48.842515 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:36:48.842487    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:41:48.862621 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:41:48.862584    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:41:48.869148 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:41:48.869124    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:46:48.887273 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:46:48.887243    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:46:48.894681 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:46:48.894659    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:51:48.911677 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:51:48.911647    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:51:48.919184 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:51:48.919162    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:53:27.164487 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:27.164455    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77cd789dcd-jfj78_e41963f3-1896-49f7-bed3-d2c3cb1f4ee3/manager/0.log"
May 28 03:53:27.268762 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:27.268734    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-68c4fbbd6f-26php_597d824b-3198-478f-bd10-0e93574496ba/postgres/0.log"
May 28 03:53:28.800983 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:28.800959    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6cb54b5c86-wcv74_fa58f5a0-4c35-48b5-b636-af3e85a31125/kuadrant-console-plugin/0.log"
May 28 03:53:29.914061 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:29.914028    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b78c488db-w5drg_42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f/kube-auth-proxy/0.log"
May 28 03:53:30.557698 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:30.557664    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6_fc6caf3d-b90f-4953-8dc5-46aca15a4139/storage-initializer/0.log"
May 28 03:53:30.563622 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:30.563604    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-7f849f6b56-ngtj6_fc6caf3d-b90f-4953-8dc5-46aca15a4139/main/0.log"
May 28 03:53:30.782914 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:30.782881    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-84db68679b-52brt_22c825cb-f657-4146-8af1-1e3505fdf3f4/storage-initializer/0.log"
May 28 03:53:30.789767 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:30.789747    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-84db68679b-52brt_22c825cb-f657-4146-8af1-1e3505fdf3f4/main/0.log"
May 28 03:53:31.006910 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:31.006880    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p_f5a91282-477a-4737-9132-c6f1f2cecdb8/storage-initializer/0.log"
May 28 03:53:31.014112 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:31.014087    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-8f8dc67b7-8jm5p_f5a91282-477a-4737-9132-c6f1f2cecdb8/main/0.log"
May 28 03:53:37.929544 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:37.929513    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s4vgs_442d53f6-d855-4957-9f81-42f46bbb30ce/global-pull-secret-syncer/0.log"
May 28 03:53:38.024826 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:38.024792    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n7qlq_d0a5d0c7-f4a0-472d-9ad2-1680f9855e3d/konnectivity-agent/0.log"
May 28 03:53:38.114432 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:38.114405    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-29.ec2.internal_fa96d698c98709a8e74d079ba7e0a5e7/haproxy/0.log"
May 28 03:53:42.605094 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:42.605068    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6cb54b5c86-wcv74_fa58f5a0-4c35-48b5-b636-af3e85a31125/kuadrant-console-plugin/0.log"
May 28 03:53:44.641478 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.641450    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-sbjxf_40bbbf19-4e88-423f-8f29-32c627762ffd/monitoring-plugin/0.log"
May 28 03:53:44.743538 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.743457    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4zm5_4f723a40-b86a-4c90-a0ae-b027eb38884b/node-exporter/0.log"
May 28 03:53:44.766288 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.766261    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4zm5_4f723a40-b86a-4c90-a0ae-b027eb38884b/kube-rbac-proxy/0.log"
May 28 03:53:44.785919 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.785889    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4zm5_4f723a40-b86a-4c90-a0ae-b027eb38884b/init-textfile/0.log"
May 28 03:53:44.882155 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.882122    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-x8mdt_545c3d5e-8f94-4b9f-a3a1-5478495dbee3/kube-rbac-proxy-main/0.log"
May 28 03:53:44.903536 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.903508    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-x8mdt_545c3d5e-8f94-4b9f-a3a1-5478495dbee3/kube-rbac-proxy-self/0.log"
May 28 03:53:44.922381 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.922351    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-x8mdt_545c3d5e-8f94-4b9f-a3a1-5478495dbee3/openshift-state-metrics/0.log"
May 28 03:53:44.953517 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.953486    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c9904fd4-14d7-4c08-a9c3-54a38676aec0/prometheus/0.log"
May 28 03:53:44.974628 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.974603    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c9904fd4-14d7-4c08-a9c3-54a38676aec0/config-reloader/0.log"
May 28 03:53:44.994784 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:44.994714    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c9904fd4-14d7-4c08-a9c3-54a38676aec0/thanos-sidecar/0.log"
May 28 03:53:45.016139 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.016112    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c9904fd4-14d7-4c08-a9c3-54a38676aec0/kube-rbac-proxy-web/0.log"
May 28 03:53:45.037287 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.037263    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c9904fd4-14d7-4c08-a9c3-54a38676aec0/kube-rbac-proxy/0.log"
May 28 03:53:45.058648 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.058621    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c9904fd4-14d7-4c08-a9c3-54a38676aec0/kube-rbac-proxy-thanos/0.log"
May 28 03:53:45.079315 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.079291    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c9904fd4-14d7-4c08-a9c3-54a38676aec0/init-config-reloader/0.log"
May 28 03:53:45.107110 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.107073    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-m8lzx_c2f27497-7ee2-46ee-b1b3-95eda95c7433/prometheus-operator/0.log"
May 28 03:53:45.128577 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.128551    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-m8lzx_c2f27497-7ee2-46ee-b1b3-95eda95c7433/kube-rbac-proxy/0.log"
May 28 03:53:45.261173 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.261070    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7899f9f89-6m59r_3d8fa66f-6d37-40d1-99a1-d53640bc50a6/thanos-query/0.log"
May 28 03:53:45.282601 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.282573    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7899f9f89-6m59r_3d8fa66f-6d37-40d1-99a1-d53640bc50a6/kube-rbac-proxy-web/0.log"
May 28 03:53:45.312869 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.312831    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7899f9f89-6m59r_3d8fa66f-6d37-40d1-99a1-d53640bc50a6/kube-rbac-proxy/0.log"
May 28 03:53:45.333875 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.333843    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7899f9f89-6m59r_3d8fa66f-6d37-40d1-99a1-d53640bc50a6/prom-label-proxy/0.log"
May 28 03:53:45.356403 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.356377    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7899f9f89-6m59r_3d8fa66f-6d37-40d1-99a1-d53640bc50a6/kube-rbac-proxy-rules/0.log"
May 28 03:53:45.376692 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:45.376668    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7899f9f89-6m59r_3d8fa66f-6d37-40d1-99a1-d53640bc50a6/kube-rbac-proxy-metrics/0.log"
May 28 03:53:46.622381 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.622335    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"]
May 28 03:53:46.622775 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.622732    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f347da-59cb-4d27-9394-b6b84ee8364b" containerName="authorino"
May 28 03:53:46.622775 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.622745    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f347da-59cb-4d27-9394-b6b84ee8364b" containerName="authorino"
May 28 03:53:46.622873 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.622850    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="00f347da-59cb-4d27-9394-b6b84ee8364b" containerName="authorino"
May 28 03:53:46.624947 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.624927    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.627377 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.627357    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q6qd7\"/\"default-dockercfg-lbjv5\""
May 28 03:53:46.627476 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.627363    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6qd7\"/\"kube-root-ca.crt\""
May 28 03:53:46.628179 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.628160    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6qd7\"/\"openshift-service-ca.crt\""
May 28 03:53:46.633957 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.633931    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"]
May 28 03:53:46.774979 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.774938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-proc\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.774979 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.774976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-lib-modules\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.775213 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.775001    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-podres\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.775213 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.775126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-sys\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.775213 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.775153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngdk\" (UniqueName: \"kubernetes.io/projected/a4baf70d-aa02-45d6-a54d-641b2c616677-kube-api-access-mngdk\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.875887 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.875790    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-sys\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.875887 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.875836    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mngdk\" (UniqueName: \"kubernetes.io/projected/a4baf70d-aa02-45d6-a54d-641b2c616677-kube-api-access-mngdk\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.876089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.875914    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-proc\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.876089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.875917    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-sys\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.876089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.875937    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-lib-modules\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.876089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.875969    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-podres\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.876089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.875971    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-proc\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.876089 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.876057    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-lib-modules\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.876271 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.876098    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4baf70d-aa02-45d6-a54d-641b2c616677-podres\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.883370 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.883332    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngdk\" (UniqueName: \"kubernetes.io/projected/a4baf70d-aa02-45d6-a54d-641b2c616677-kube-api-access-mngdk\") pod \"perf-node-gather-daemonset-mm5x4\" (UID: \"a4baf70d-aa02-45d6-a54d-641b2c616677\") " pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:46.935653 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:46.935620    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:47.042298 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:47.042223    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/1.log"
May 28 03:53:47.048574 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:47.048543    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-bqzm4_04dfd913-171c-498e-b7ae-d73e936b328f/console-operator/2.log"
May 28 03:53:47.063310 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:47.063288    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"]
May 28 03:53:47.065574 ip-10-0-141-29 kubenswrapper[2569]: W0528 03:53:47.065544    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda4baf70d_aa02_45d6_a54d_641b2c616677.slice/crio-d0a6c9d5d71173d82059f9c35077cf0b16f0b57bafef1e6c4c7328d69cbe8a2d WatchSource:0}: Error finding container d0a6c9d5d71173d82059f9c35077cf0b16f0b57bafef1e6c4c7328d69cbe8a2d: Status 404 returned error can't find the container with id d0a6c9d5d71173d82059f9c35077cf0b16f0b57bafef1e6c4c7328d69cbe8a2d
May 28 03:53:47.067131 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:47.067117    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 03:53:47.295252 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:47.295211    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4" event={"ID":"a4baf70d-aa02-45d6-a54d-641b2c616677","Type":"ContainerStarted","Data":"a9f6300f365d461cd81a8962d35d9c945eea44b91dfbe1b5fe8ead2c8a7fead8"}
May 28 03:53:47.295252 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:47.295252    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4" event={"ID":"a4baf70d-aa02-45d6-a54d-641b2c616677","Type":"ContainerStarted","Data":"d0a6c9d5d71173d82059f9c35077cf0b16f0b57bafef1e6c4c7328d69cbe8a2d"}
May 28 03:53:47.295480 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:47.295332    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:47.313337 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:47.313283    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4" podStartSLOduration=1.313266681 podStartE2EDuration="1.313266681s" podCreationTimestamp="2026-05-28 03:53:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 03:53:47.31086338 +0000 UTC m=+2219.026384084" watchObservedRunningTime="2026-05-28 03:53:47.313266681 +0000 UTC m=+2219.028787372"
May 28 03:53:47.615453 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:47.615425    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-rgqh9_b3422cfc-ac5d-49d9-9f8e-0f5b73d68b30/download-server/0.log"
May 28 03:53:48.886594 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:48.886570    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9mc89_eb53683e-628a-4184-9360-1ea2c224534f/dns/0.log"
May 28 03:53:48.909694 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:48.909669    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9mc89_eb53683e-628a-4184-9360-1ea2c224534f/kube-rbac-proxy/0.log"
May 28 03:53:49.029395 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:49.029372    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54fkb_c9e1ada3-6207-4ebb-84d1-edf6e12f7311/dns-node-resolver/0.log"
May 28 03:53:49.550645 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:49.550613    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84594f8d74-nlpnd_2a3070b1-0be1-4c5d-8014-ed51a22d0e18/registry/0.log"
May 28 03:53:49.591604 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:49.591574    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jjnl2_64beddb4-335a-4ddb-bc37-07555ed4a8f7/node-ca/0.log"
May 28 03:53:50.648575 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:50.648536    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b78c488db-w5drg_42b7ec75-10d2-43b5-92a0-0d7d1ab6ca4f/kube-auth-proxy/0.log"
May 28 03:53:51.420576 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:51.420545    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b4qwv_8a1688cd-5f8c-4782-9652-9f0ab1e0206d/serve-healthcheck-canary/0.log"
May 28 03:53:51.988657 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:51.988633    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-78rv9_ddc3070d-dd0b-4926-9bb7-d4ed87f035de/kube-rbac-proxy/0.log"
May 28 03:53:52.027218 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:52.027195    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-78rv9_ddc3070d-dd0b-4926-9bb7-d4ed87f035de/exporter/0.log"
May 28 03:53:52.050626 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:52.050599    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-78rv9_ddc3070d-dd0b-4926-9bb7-d4ed87f035de/extractor/0.log"
May 28 03:53:53.308908 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:53.308855    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q6qd7/perf-node-gather-daemonset-mm5x4"
May 28 03:53:54.503240 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:54.503212    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77cd789dcd-jfj78_e41963f3-1896-49f7-bed3-d2c3cb1f4ee3/manager/0.log"
May 28 03:53:54.524247 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:54.524225    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-68c4fbbd6f-26php_597d824b-3198-478f-bd10-0e93574496ba/postgres/0.log"
May 28 03:53:55.835866 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:53:55.835838    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8546f974ff-rx48j_bb4d1987-2d52-4d3b-b6dd-5b1511bed63b/manager/0.log"
May 28 03:54:01.857885 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:01.857853    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mj6lt_8d2fdca3-5c87-470f-948f-b6b2b2df48ae/kube-multus-additional-cni-plugins/0.log"
May 28 03:54:01.880365 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:01.880313    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mj6lt_8d2fdca3-5c87-470f-948f-b6b2b2df48ae/egress-router-binary-copy/0.log"
May 28 03:54:01.903800 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:01.903772    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mj6lt_8d2fdca3-5c87-470f-948f-b6b2b2df48ae/cni-plugins/0.log"
May 28 03:54:01.923724 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:01.923699    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mj6lt_8d2fdca3-5c87-470f-948f-b6b2b2df48ae/bond-cni-plugin/0.log"
May 28 03:54:01.943772 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:01.943749    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mj6lt_8d2fdca3-5c87-470f-948f-b6b2b2df48ae/routeoverride-cni/0.log"
May 28 03:54:01.964268 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:01.964245    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mj6lt_8d2fdca3-5c87-470f-948f-b6b2b2df48ae/whereabouts-cni-bincopy/0.log"
May 28 03:54:01.983802 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:01.983777    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mj6lt_8d2fdca3-5c87-470f-948f-b6b2b2df48ae/whereabouts-cni/0.log"
May 28 03:54:02.043397 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:02.043366    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zhtxl_db69a13b-da03-43b3-af25-a070cfe57890/kube-multus/0.log"
May 28 03:54:02.062637 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:02.062601    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4z6ww_1025fde6-0949-48f6-af56-3ecd15e95ff7/network-metrics-daemon/0.log"
May 28 03:54:02.083637 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:02.083613    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4z6ww_1025fde6-0949-48f6-af56-3ecd15e95ff7/kube-rbac-proxy/0.log"
May 28 03:54:02.985250 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:02.985221    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qfb9_3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de/ovn-controller/0.log"
May 28 03:54:03.012171 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:03.012138    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qfb9_3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de/ovn-acl-logging/0.log"
May 28 03:54:03.030703 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:03.030674    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qfb9_3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de/kube-rbac-proxy-node/0.log"
May 28 03:54:03.051315 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:03.051275    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qfb9_3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de/kube-rbac-proxy-ovn-metrics/0.log"
May 28 03:54:03.071516 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:03.071494    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qfb9_3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de/northd/0.log"
May 28 03:54:03.091065 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:03.091043    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qfb9_3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de/nbdb/0.log"
May 28 03:54:03.110741 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:03.110714    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qfb9_3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de/sbdb/0.log"
May 28 03:54:03.216229 ip-10-0-141-29 kubenswrapper[2569]: I0528 03:54:03.216197    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qfb9_3b1cabe6-c7fe-4a0c-8b4d-24ca3d2313de/ovnkube-controller/0.log"