May 28 15:08:09.870924 ip-10-0-130-203 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 28 15:08:09.870936 ip-10-0-130-203 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 28 15:08:09.870943 ip-10-0-130-203 systemd[1]: kubelet.service: Failed with result 'resources'.
May 28 15:08:09.871182 ip-10-0-130-203 systemd[1]: Failed to start Kubernetes Kubelet.
May 28 15:08:20.104121 ip-10-0-130-203 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 28 15:08:20.104142 ip-10-0-130-203 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot fccf8e23c2ff402db0f173740c3b50f7 --
May 28 15:10:47.568547 ip-10-0-130-203 systemd[1]: Starting Kubernetes Kubelet...
May 28 15:10:48.067801 ip-10-0-130-203 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 15:10:48.067801 ip-10-0-130-203 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 28 15:10:48.067801 ip-10-0-130-203 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 15:10:48.067801 ip-10-0-130-203 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 28 15:10:48.067801 ip-10-0-130-203 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 15:10:48.069743 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.069654    2582 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 28 15:10:48.077550 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077523    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 15:10:48.077550 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077543    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 15:10:48.077550 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077547    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 15:10:48.077550 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077550    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 15:10:48.077550 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077553    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 15:10:48.077550 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077555    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 15:10:48.077550 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077559    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077563    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077565    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077568    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077571    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077573    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077576    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077578    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077581    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077584    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077587    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077589    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077592    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077594    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077598    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077603    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077606    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077609    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077611    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077616    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 15:10:48.077793 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077619    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077623    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077627    2582 feature_gate.go:328] unrecognized feature gate: Example
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077630    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077634    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077637    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077639    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077642    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077644    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077648    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077651    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077653    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077655    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077658    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077662    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077664    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077667    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077669    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077672    2582 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 15:10:48.078296 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077675    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077677    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077679    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077682    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077684    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077687    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077689    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077692    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077694    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077696    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077700    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077702    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077705    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077707    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077709    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077711    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077714    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077716    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077719    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 15:10:48.078763 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077721    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077723    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077726    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077728    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077730    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077733    2582 feature_gate.go:328] unrecognized feature gate: Example2
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077735    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077738    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077741    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077744    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077747    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077749    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077752    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077754    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077758    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077760    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077762    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077765    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077767    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077770    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 15:10:48.079219 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077772    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.077774    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078169    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078175    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078178    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078182    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078186    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078189    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078192    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078195    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078198    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078200    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078203    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078205    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078208    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078211    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078213    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078215    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078218    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078220    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 15:10:48.079716 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078224    2582 feature_gate.go:328] unrecognized feature gate: Example2
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078227    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078230    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078233    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078235    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078238    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078242    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078244    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078247    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078249    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078251    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078254    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078256    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078259    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078261    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078264    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078266    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078268    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078271    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 15:10:48.080199 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078273    2582 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078275    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078278    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078280    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078283    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078285    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078288    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078290    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078292    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078296    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078299    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078302    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078305    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078308    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078311    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078313    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078316    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078318    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078321    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 15:10:48.080725 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078324    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078336    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078338    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078341    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078344    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078346    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078349    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078354    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078357    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078360    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078362    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078365    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078367    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078370    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078372    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078374    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078377    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078379    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078382    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078384    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 15:10:48.081194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078386    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078389    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078391    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078394    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078397    2582 feature_gate.go:328] unrecognized feature gate: Example
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078400    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078402    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078405    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078408    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.078410    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078495    2582 flags.go:64] FLAG: --address="0.0.0.0"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078513    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078521    2582 flags.go:64] FLAG: --anonymous-auth="true"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078527    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078533    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078536    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078540    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078545    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078548    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078551    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078554    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078557    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 28 15:10:48.081693 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078560    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078563    2582 flags.go:64] FLAG: --cgroup-root=""
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078566    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078569    2582 flags.go:64] FLAG: --client-ca-file=""
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078571    2582 flags.go:64] FLAG: --cloud-config=""
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078574    2582 flags.go:64] FLAG: --cloud-provider="external"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078577    2582 flags.go:64] FLAG: --cluster-dns="[]"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078581    2582 flags.go:64] FLAG: --cluster-domain=""
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078584    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078587    2582 flags.go:64] FLAG: --config-dir=""
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078590    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078593    2582 flags.go:64] FLAG: --container-log-max-files="5"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078597    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078600    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078603    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078607    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078610    2582 flags.go:64] FLAG: --contention-profiling="false"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078613    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078616    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078619    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078621    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078625    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078628    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078631    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078634    2582 flags.go:64] FLAG: --enable-load-reader="false"
May 28 15:10:48.082233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078637    2582 flags.go:64] FLAG: --enable-server="true"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078640    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078644    2582 flags.go:64] FLAG: --event-burst="100"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078647    2582 flags.go:64] FLAG: --event-qps="50"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078650    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078653    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078656    2582 flags.go:64] FLAG: --eviction-hard=""
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078660    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078662    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078665    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078668    2582 flags.go:64] FLAG: --eviction-soft=""
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078672    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078674    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078677    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078680    2582 flags.go:64] FLAG: --experimental-mounter-path=""
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078683    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078686    2582 flags.go:64] FLAG: --fail-swap-on="true"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078689    2582 flags.go:64] FLAG: --feature-gates=""
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078693    2582 flags.go:64] FLAG: --file-check-frequency="20s"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078695    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078698    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078702    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078705    2582 flags.go:64] FLAG: --healthz-port="10248"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078707    2582 flags.go:64] FLAG: --help="false"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078711    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-130-203.ec2.internal"
May 28 15:10:48.082823 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078713    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078716    2582 flags.go:64] FLAG: --http-check-frequency="20s"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078719    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078722    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078726    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078728    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078731    2582 flags.go:64] FLAG: --image-service-endpoint=""
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078734    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078740    2582 flags.go:64] FLAG: --kube-api-burst="100"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078743    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078746    2582 flags.go:64] FLAG: --kube-api-qps="50"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078749    2582 flags.go:64] FLAG: --kube-reserved=""
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078751    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078754    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078757    2582 flags.go:64] FLAG: --kubelet-cgroups=""
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078760    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078762    2582 flags.go:64] FLAG: --lock-file=""
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078765    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078773    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078776    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078782    2582 flags.go:64] FLAG: --log-json-split-stream="false"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078785    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078788    2582 flags.go:64] FLAG: --log-text-split-stream="false"
May 28 15:10:48.083413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078791    2582 flags.go:64] FLAG: --logging-format="text"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078794    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078797    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078800    2582 flags.go:64] FLAG: --manifest-url=""
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078803    2582 flags.go:64] FLAG: --manifest-url-header=""
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078807    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078810    2582 flags.go:64] FLAG: --max-open-files="1000000"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078814    2582 flags.go:64] FLAG: --max-pods="110"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078817    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078820    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078822    2582 flags.go:64] FLAG: --memory-manager-policy="None"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078825    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078828    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078831    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078834    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078841    2582 flags.go:64] FLAG: --node-status-max-images="50"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078844    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078846    2582 flags.go:64] FLAG: --oom-score-adj="-999"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078850    2582 flags.go:64] FLAG: --pod-cidr=""
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078853    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078858    2582 flags.go:64] FLAG: --pod-manifest-path=""
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078860    2582 flags.go:64] FLAG: --pod-max-pids="-1"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078863    2582 flags.go:64] FLAG: --pods-per-core="0"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078866    2582 flags.go:64] FLAG: --port="10250"
May 28 15:10:48.083950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078869    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078872    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06deabcc45223594f"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078875    2582 flags.go:64] FLAG: --qos-reserved=""
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078879    2582 flags.go:64] FLAG: --read-only-port="10255"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078881    2582 flags.go:64] FLAG: --register-node="true"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078884    2582 flags.go:64] FLAG: --register-schedulable="true"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078887    2582 flags.go:64] FLAG: --register-with-taints=""
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078890    2582 flags.go:64] FLAG: --registry-burst="10"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078893    2582 flags.go:64] FLAG: --registry-qps="5"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078896    2582 flags.go:64] FLAG: --reserved-cpus=""
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078899    2582 flags.go:64] FLAG: --reserved-memory=""
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078902    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078906    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078908    2582 flags.go:64] FLAG: --rotate-certificates="false"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078911    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078914    2582 flags.go:64] FLAG: --runonce="false"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078917    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078920    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078922    2582 flags.go:64] FLAG: --seccomp-default="false"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078925    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078928    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078931    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078934    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078937    2582 flags.go:64] FLAG: --storage-driver-password="root"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078939    2582 flags.go:64] FLAG: --storage-driver-secure="false"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078942    2582 flags.go:64] FLAG: --storage-driver-table="stats"
May 28 15:10:48.084560 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078945    2582 flags.go:64] FLAG: --storage-driver-user="root"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078948    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078951    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078954    2582 flags.go:64] FLAG: --system-cgroups=""
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078956    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078961    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078964    2582 flags.go:64] FLAG: --tls-cert-file=""
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078967    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078971    2582 flags.go:64] FLAG: --tls-min-version=""
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078975    2582 flags.go:64] FLAG: --tls-private-key-file=""
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078978    2582 flags.go:64] FLAG: --topology-manager-policy="none"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078981    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078983    2582 flags.go:64] FLAG: --topology-manager-scope="container"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078986    2582 flags.go:64] FLAG: --v="2"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078990    2582 flags.go:64] FLAG: --version="false"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078994    2582 flags.go:64] FLAG: --vmodule=""
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.078998    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.079002    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079087    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079091    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079094    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079096    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079099    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 15:10:48.085182 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079101    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079104    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079106    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079109    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079111    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079114    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079117    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079133    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079135    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079139    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079141    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079145    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079147    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079150    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079152    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079155    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079157    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079160    2582 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079164    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079167    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 15:10:48.085713 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079170    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079172    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079175    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079177    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079180    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079183    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079185    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079187    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079190    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079193    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079195    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079198    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079200    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079203    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079206    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079208    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079211    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079213    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079216    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079219    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 15:10:48.086238 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079221    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079224    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079227    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079229    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079233    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079235    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079237    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079240    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079242    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079245    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079249    2582 feature_gate.go:328] unrecognized feature gate: Example
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079251    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079254    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079256    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079261    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079264    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079266    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079268    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079271    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079273    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 15:10:48.086743 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079276    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079278    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079281    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079283    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079286    2582 feature_gate.go:328] unrecognized feature gate: Example2
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079288    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079291    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079293    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079295    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079299    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079303    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079306    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079309    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079312    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079316    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079318    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079321    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079324    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079327    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 15:10:48.087236 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079330    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 15:10:48.087738 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.079332    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 15:10:48.087738 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.080092    2582 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 15:10:48.088497 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.088477    2582 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 28 15:10:48.088533 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.088499    2582 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 28 15:10:48.088560 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088547    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 15:10:48.088560 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088553    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 15:10:48.088560 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088556    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 15:10:48.088560 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088559    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088563    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088568    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088572    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088574    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088577    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088580    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088582    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088585    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088587    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088590    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088592    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088595    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088598    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088601    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088603    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088606    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088609    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088611    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 15:10:48.088658 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088614    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088616    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088619    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088621    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088625    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088629    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088632    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088635    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088637    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088641    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088644    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088646    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088649    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088651    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088654    2582 feature_gate.go:328] unrecognized feature gate: Example2
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088658    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088661    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088663    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088666    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 15:10:48.089107 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088669    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088672    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088674    2582 feature_gate.go:328] unrecognized feature gate: Example
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088677    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088680    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088682    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088684    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088687    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088689    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088692    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088694    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088697    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088699    2582 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088701    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088704    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088706    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088709    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088712    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088714    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088717    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 15:10:48.089625 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088719    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088722    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088724    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088727    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088730    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088732    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088735    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088738    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088740    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088743    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088745    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088748    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088751    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088753    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088756    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088758    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088761    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088763    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088766    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088768    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 15:10:48.090098 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088772    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088774    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088777    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088779    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088781    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.088786    2582 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088884    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088889    2582 feature_gate.go:328] unrecognized feature gate: Example
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088892    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088895    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088897    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088900    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088903    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088906    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088908    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088911    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 15:10:48.090583 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088914    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088917    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088920    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088923    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088926    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088929    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088931    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088935    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088938    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088941    2582 feature_gate.go:328] unrecognized feature gate: Example2
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088944    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088947    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088950    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088953    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088955    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088957    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088960    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088962    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088965    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 15:10:48.090968 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088967    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088970    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088972    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088975    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088977    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088980    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088982    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088985    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088987    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088990    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088992    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088995    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.088997    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089000    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089002    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089005    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089008    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089012    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089015    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 15:10:48.091437 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089018    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089020    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089023    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089025    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089028    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089030    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089033    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089035    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089037    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089040    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089042    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089045    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089047    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089050    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089052    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089054    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089056    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089059    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089061    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089064    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 15:10:48.091921 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089066    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089068    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089071    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089073    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089075    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089078    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089080    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089083    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089086    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089089    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089093    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089095    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089097    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089100    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089102    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089105    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089107    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 15:10:48.092406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:48.089109    2582 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 15:10:48.092808 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.089114    2582 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 15:10:48.092808 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.089954    2582 server.go:962] "Client rotation is on, will bootstrap in background"
May 28 15:10:48.092808 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.092201    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 28 15:10:48.093402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.093390    2582 server.go:1019] "Starting client certificate rotation"
May 28 15:10:48.093502 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.093487    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 15:10:48.093531 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.093523    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 15:10:48.122499 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.122480    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 15:10:48.125990 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.125966    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 15:10:48.140637 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.140621    2582 log.go:25] "Validated CRI v1 runtime API"
May 28 15:10:48.146677 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.146664    2582 log.go:25] "Validated CRI v1 image API"
May 28 15:10:48.149756 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.149736    2582 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 28 15:10:48.152561 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.152543    2582 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 15:10:48.153951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.153928    2582 fs.go:135] Filesystem UUIDs: map[64d97d2a-2770-4063-90c1-c02822cd1075:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 aebc129a-979f-41bf-bd78-3e79413aff4e:/dev/nvme0n1p4]
May 28 15:10:48.154030 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.153949    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 28 15:10:48.158064 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.157952    2582 manager.go:217] Machine: {Timestamp:2026-05-28 15:10:48.157485561 +0000 UTC m=+0.456555065 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198770 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b8e8fd1079679806a10ec69bb4eb1 SystemUUID:ec2b8e8f-d107-9679-806a-10ec69bb4eb1 BootID:fccf8e23-c2ff-402d-b0f1-73740c3b50f7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d8:32:30:ba:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d8:32:30:ba:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:80:64:3e:cd:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 28 15:10:48.158798 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.158787    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 28 15:10:48.158917 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.158906    2582 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 28 15:10:48.161031 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.161010    2582 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 28 15:10:48.161184 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.161034    2582 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-203.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 28 15:10:48.161227 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.161193    2582 topology_manager.go:138] "Creating topology manager with none policy"
May 28 15:10:48.161227 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.161202    2582 container_manager_linux.go:306] "Creating device plugin manager"
May 28 15:10:48.161227 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.161210    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 15:10:48.162225 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.162214    2582 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 15:10:48.164127 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.164108    2582 state_mem.go:36] "Initialized new in-memory state store"
May 28 15:10:48.164245 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.164237    2582 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 28 15:10:48.166958 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.166947    2582 kubelet.go:491] "Attempting to sync node with API server"
May 28 15:10:48.166991 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.166962    2582 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 28 15:10:48.166991 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.166975    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 28 15:10:48.166991 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.166984    2582 kubelet.go:397] "Adding apiserver pod source"
May 28 15:10:48.167060 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.166993    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 28 15:10:48.168224 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.168211    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 15:10:48.168277 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.168236    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 15:10:48.171185 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.171159    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6n6r2"
May 28 15:10:48.172027 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.172006    2582 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 28 15:10:48.174980 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.174966    2582 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 28 15:10:48.179177 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179161    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 28 15:10:48.179268 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179185    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 28 15:10:48.179268 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179195    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 28 15:10:48.179268 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179203    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 28 15:10:48.179268 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179217    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 28 15:10:48.179268 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179233    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 28 15:10:48.179268 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179242    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 28 15:10:48.179268 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179251    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 28 15:10:48.179268 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179263    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 28 15:10:48.179500 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179272    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 28 15:10:48.179500 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179286    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 28 15:10:48.179580 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.179570    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 28 15:10:48.180627 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.180612    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 28 15:10:48.180705 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.180637    2582 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 28 15:10:48.181638 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.181618    2582 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-203.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 28 15:10:48.182311 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.182270    2582 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 28 15:10:48.182402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.182382    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6n6r2"
May 28 15:10:48.184013 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.183998    2582 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 28 15:10:48.184089 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.184046    2582 server.go:1295] "Started kubelet"
May 28 15:10:48.184172 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.184143    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 28 15:10:48.184219 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.184160    2582 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 28 15:10:48.184274 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.184230    2582 server_v1.go:47] "podresources" method="list" useActivePods=true
May 28 15:10:48.184765 ip-10-0-130-203 systemd[1]: Started Kubernetes Kubelet.
May 28 15:10:48.185303 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.185243    2582 server.go:317] "Adding debug handlers to kubelet server"
May 28 15:10:48.186292 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.186277    2582 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 28 15:10:48.193315 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.193292    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 28 15:10:48.193930 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.193911    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 28 15:10:48.194032 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.194019    2582 volume_manager.go:295] "The desired_state_of_world populator starts"
May 28 15:10:48.194032 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.194029    2582 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 28 15:10:48.194165 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.194085    2582 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 28 15:10:48.194237 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.194223    2582 reconstruct.go:97] "Volume reconstruction finished"
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.194591    2582 reconciler.go:26] "Reconciler: start to sync state"
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.194961    2582 factory.go:55] Registering systemd factory
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.195055    2582 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-203.ec2.internal" not found
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.195064    2582 factory.go:223] Registration of the systemd container factory successfully
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.195806    2582 factory.go:153] Registering CRI-O factory
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.195821    2582 factory.go:223] Registration of the crio container factory successfully
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.195851    2582 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.195874    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.195900    2582 factory.go:103] Registering Raw factory
May 28 15:10:48.196019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.195915    2582 manager.go:1196] Started watching for new ooms in manager
May 28 15:10:48.196577 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.196298    2582 manager.go:319] Starting recovery of all containers
May 28 15:10:48.197090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.197067    2582 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 15:10:48.197657 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.197620    2582 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 28 15:10:48.198950 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.198930    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-203.ec2.internal\" not found" node="ip-10-0-130-203.ec2.internal"
May 28 15:10:48.205665 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.205649    2582 manager.go:324] Recovery completed
May 28 15:10:48.209392 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.209373    2582 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-203.ec2.internal" not found
May 28 15:10:48.210510 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.210495    2582 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 15:10:48.212548 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.212534    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientMemory"
May 28 15:10:48.212598 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.212560    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasNoDiskPressure"
May 28 15:10:48.212598 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.212570    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientPID"
May 28 15:10:48.213063 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.213048    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 28 15:10:48.213063 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.213061    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 28 15:10:48.213193 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.213077    2582 state_mem.go:36] "Initialized new in-memory state store"
May 28 15:10:48.216985 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.216968    2582 policy_none.go:49] "None policy: Start"
May 28 15:10:48.216985 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.216984    2582 memory_manager.go:186] "Starting memorymanager" policy="None"
May 28 15:10:48.217081 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.216993    2582 state_mem.go:35] "Initializing new in-memory state store"
May 28 15:10:48.251690 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.251669    2582 manager.go:341] "Starting Device Plugin manager"
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.251697    2582 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.251706    2582 server.go:85] "Starting device plugin registration server"
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.251909    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.251919    2582 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.252028    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.252089    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.252097    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.252673    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.252707    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:48.268192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.265946    2582 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-203.ec2.internal" not found
May 28 15:10:48.286098 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.286070    2582 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 28 15:10:48.287307 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.287293    2582 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 28 15:10:48.287350 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.287313    2582 status_manager.go:230] "Starting to sync pod status with apiserver"
May 28 15:10:48.287350 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.287330    2582 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 28 15:10:48.287350 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.287336    2582 kubelet.go:2451] "Starting kubelet main sync loop"
May 28 15:10:48.287444 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.287420    2582 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 28 15:10:48.293541 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.293524    2582 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 15:10:48.352491 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.352403    2582 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 15:10:48.354469 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.354454    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientMemory"
May 28 15:10:48.354538 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.354484    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasNoDiskPressure"
May 28 15:10:48.354538 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.354493    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientPID"
May 28 15:10:48.354538 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.354517    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-203.ec2.internal"
May 28 15:10:48.364035 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.364019    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-203.ec2.internal"
May 28 15:10:48.364084 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.364040    2582 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-203.ec2.internal\": node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:48.381803 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.381786    2582 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:48.387881 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.387860    2582 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal"]
May 28 15:10:48.387953 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.387942    2582 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 15:10:48.388771 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.388760    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientMemory"
May 28 15:10:48.388813 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.388789    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasNoDiskPressure"
May 28 15:10:48.388813 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.388800    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientPID"
May 28 15:10:48.389775 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.389763    2582 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 15:10:48.389935 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.389922    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.389977 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.389948    2582 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 15:10:48.390757 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.390730    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientMemory"
May 28 15:10:48.390757 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.390742    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientMemory"
May 28 15:10:48.390757 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.390758    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasNoDiskPressure"
May 28 15:10:48.390968 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.390764    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasNoDiskPressure"
May 28 15:10:48.390968 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.390773    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientPID"
May 28 15:10:48.390968 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.390773    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientPID"
May 28 15:10:48.391640 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.391625    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.391689 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.391656    2582 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 15:10:48.392294 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.392281    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientMemory"
May 28 15:10:48.392355 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.392309    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasNoDiskPressure"
May 28 15:10:48.392355 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.392324    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeHasSufficientPID"
May 28 15:10:48.395912 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.395891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6642e2f3392820093d8ef926280a2cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal\" (UID: \"b6642e2f3392820093d8ef926280a2cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.395959 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.395929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6642e2f3392820093d8ef926280a2cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal\" (UID: \"b6642e2f3392820093d8ef926280a2cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.395998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.395982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1691e4b38b2da49454ca8518fd1a04cb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-203.ec2.internal\" (UID: \"1691e4b38b2da49454ca8518fd1a04cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.405604 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.405589    2582 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-203.ec2.internal\" not found" node="ip-10-0-130-203.ec2.internal"
May 28 15:10:48.409275 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.409262    2582 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-203.ec2.internal\" not found" node="ip-10-0-130-203.ec2.internal"
May 28 15:10:48.482669 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.482637    2582 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:48.496421 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.496401    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6642e2f3392820093d8ef926280a2cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal\" (UID: \"b6642e2f3392820093d8ef926280a2cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.496490 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.496358    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6642e2f3392820093d8ef926280a2cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal\" (UID: \"b6642e2f3392820093d8ef926280a2cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.496490 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.496462    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1691e4b38b2da49454ca8518fd1a04cb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-203.ec2.internal\" (UID: \"1691e4b38b2da49454ca8518fd1a04cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.496490 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.496478    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6642e2f3392820093d8ef926280a2cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal\" (UID: \"b6642e2f3392820093d8ef926280a2cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.496583 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.496506    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6642e2f3392820093d8ef926280a2cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal\" (UID: \"b6642e2f3392820093d8ef926280a2cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.496583 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.496536    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1691e4b38b2da49454ca8518fd1a04cb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-203.ec2.internal\" (UID: \"1691e4b38b2da49454ca8518fd1a04cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.583766 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.583737    2582 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:48.684493 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.684459    2582 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:48.707977 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.707945    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.711677 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:48.711662    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal"
May 28 15:10:48.785223 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.785176    2582 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:48.885741 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.885706    2582 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:48.986371 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:48.986160    2582 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-203.ec2.internal\" not found"
May 28 15:10:49.072893 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.072865    2582 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 15:10:49.093155 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.092826    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 28 15:10:49.093155 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.092961    2582 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 15:10:49.093155 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.092994    2582 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 15:10:49.093155 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.093027    2582 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 15:10:49.095660 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.095641    2582 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal"
May 28 15:10:49.117939 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.117919    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 15:10:49.120010 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.119997    2582 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal"
May 28 15:10:49.134279 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.134258    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 15:10:49.167430 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.167400    2582 apiserver.go:52] "Watching apiserver"
May 28 15:10:49.178252 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.178232    2582 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 28 15:10:49.179092 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.179071    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-qd8qz","kube-system/konnectivity-agent-9pvmp","kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dcgvr","openshift-dns/node-resolver-sd8jw","openshift-network-diagnostics/network-check-target-j5kk5","openshift-ovn-kubernetes/ovnkube-node-tx2n2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb","openshift-image-registry/node-ca-bjtv8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal","openshift-multus/multus-additional-cni-plugins-lpq57","openshift-multus/multus-b9mt2","openshift-multus/network-metrics-daemon-9lnwm"]
May 28 15:10:49.181393 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.181370    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.184250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.184192    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-27 15:05:48 +0000 UTC" deadline="2028-02-20 16:10:16.250186452 +0000 UTC"
May 28 15:10:49.184250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.184226    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15192h59m27.065964143s"
May 28 15:10:49.184418 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.184288    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 28 15:10:49.184821 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.184472    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 28 15:10:49.184821 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.184473    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 28 15:10:49.184821 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.184481    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 28 15:10:49.184821 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.184690    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 28 15:10:49.185061 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.185044    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xrm7w\""
May 28 15:10:49.185163 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.185147    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 28 15:10:49.187369 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.187349    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:10:49.189344 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.189327    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 28 15:10:49.189437 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.189351    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 28 15:10:49.189503 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.189489    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.189553 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.189541    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gqbv8\""
May 28 15:10:49.189602 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.189561    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.191524 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.191508    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 28 15:10:49.191604 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.191555    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:49.191657 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.191600    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 28 15:10:49.191657 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.191624    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:10:49.191744 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.191687    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ps5p7\""
May 28 15:10:49.191785 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.191770    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 28 15:10:49.191932 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.191916    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lhm88\""
May 28 15:10:49.192005 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.191992    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 28 15:10:49.193382 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.193367    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 28 15:10:49.193802 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.193787    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.195961 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.195943    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 28 15:10:49.196157 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.196110    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 28 15:10:49.196239 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.196116    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 28 15:10:49.196296 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.196248    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2ft2j\""
May 28 15:10:49.198996 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.198978    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.199103 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.199089    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.201220 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs8vm\" (UniqueName: \"kubernetes.io/projected/085514bb-7b92-43e7-b961-c3d99f6cd3a5-kube-api-access-qs8vm\") pod \"iptables-alerter-qd8qz\" (UID: \"085514bb-7b92-43e7-b961-c3d99f6cd3a5\") " pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.201330 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-etc-openvswitch\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.201330 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-node-log\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.201330 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-run-ovn-kubernetes\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.201330 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201330    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-modprobe-d\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-lib-modules\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201385    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc64f60e-b2fb-46c2-b608-93ed763fbdc6-konnectivity-ca\") pod \"konnectivity-agent-9pvmp\" (UID: \"dc64f60e-b2fb-46c2-b608-93ed763fbdc6\") " pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/085514bb-7b92-43e7-b961-c3d99f6cd3a5-iptables-alerter-script\") pod \"iptables-alerter-qd8qz\" (UID: \"085514bb-7b92-43e7-b961-c3d99f6cd3a5\") " pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201427    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/085514bb-7b92-43e7-b961-c3d99f6cd3a5-host-slash\") pod \"iptables-alerter-qd8qz\" (UID: \"085514bb-7b92-43e7-b961-c3d99f6cd3a5\") " pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-kubelet\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201451    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-slash\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-run-ovn\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d8b93da-b2de-4a1c-b264-604d7329745f-ovnkube-config\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.201594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f11838e8-88b0-4035-85e4-99fd5e9026c3-tmp-dir\") pod \"node-resolver-sd8jw\" (UID: \"f11838e8-88b0-4035-85e4-99fd5e9026c3\") " pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d8b93da-b2de-4a1c-b264-604d7329745f-ovn-node-metrics-cert\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-sysctl-d\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-sys\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-var-lib-kubelet\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81941efe-61e4-45de-9f42-87037c076e64-etc-tuned\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-sys-fs\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f4cz\" (UniqueName: \"kubernetes.io/projected/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-kube-api-access-8f4cz\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc64f60e-b2fb-46c2-b608-93ed763fbdc6-agent-certs\") pod \"konnectivity-agent-9pvmp\" (UID: \"dc64f60e-b2fb-46c2-b608-93ed763fbdc6\") " pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74c6m\" (UniqueName: \"kubernetes.io/projected/f11838e8-88b0-4035-85e4-99fd5e9026c3-kube-api-access-74c6m\") pod \"node-resolver-sd8jw\" (UID: \"f11838e8-88b0-4035-85e4-99fd5e9026c3\") " pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-var-lib-openvswitch\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-log-socket\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-cni-bin\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-systemd\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.201991    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-host\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202026    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnmdv\""
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn8tg\" (UniqueName: \"kubernetes.io/projected/81941efe-61e4-45de-9f42-87037c076e64-kube-api-access-sn8tg\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.202146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202068    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202072    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-systemd-units\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202087    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cbpfk\""
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202109    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfmlf\" (UniqueName: \"kubernetes.io/projected/3d8b93da-b2de-4a1c-b264-604d7329745f-kube-api-access-rfmlf\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-device-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-run-netns\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-run-systemd\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-sysctl-conf\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f11838e8-88b0-4035-85e4-99fd5e9026c3-hosts-file\") pod \"node-resolver-sd8jw\" (UID: \"f11838e8-88b0-4035-85e4-99fd5e9026c3\") " pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202268    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-run-openvswitch\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202301    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202322    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202313    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d8b93da-b2de-4a1c-b264-604d7329745f-env-overrides\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-kubernetes\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-socket-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.202779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-cni-netd\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.203361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d8b93da-b2de-4a1c-b264-604d7329745f-ovnkube-script-lib\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.203361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-sysconfig\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.203361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-run\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.203361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81941efe-61e4-45de-9f42-87037c076e64-tmp\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.203361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.202536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-registration-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.204037 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.204023    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.204646 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.204629    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 28 15:10:49.204726 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.204662    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 28 15:10:49.205521 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.205354    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 28 15:10:49.205521 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.205358    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 28 15:10:49.205521 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.205362    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 28 15:10:49.205805 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.205762    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9g5f5\""
May 28 15:10:49.206183 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.206167    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 28 15:10:49.206248 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.206186    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tmn6w\""
May 28 15:10:49.206555 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.206540    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:49.206624 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.206591    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:10:49.207513 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.207492    2582 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 15:10:49.220194 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.220152    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1691e4b38b2da49454ca8518fd1a04cb.slice/crio-f10e92e19e2238f921d85cb1b31e0e182b3750887ecb4ef99ed5e1a395a13d64 WatchSource:0}: Error finding container f10e92e19e2238f921d85cb1b31e0e182b3750887ecb4ef99ed5e1a395a13d64: Status 404 returned error can't find the container with id f10e92e19e2238f921d85cb1b31e0e182b3750887ecb4ef99ed5e1a395a13d64
May 28 15:10:49.220398 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.220379    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6642e2f3392820093d8ef926280a2cb.slice/crio-8802dd018f36623134267dc6c520bd018ac5b8d59209ecd39bec294befec5667 WatchSource:0}: Error finding container 8802dd018f36623134267dc6c520bd018ac5b8d59209ecd39bec294befec5667: Status 404 returned error can't find the container with id 8802dd018f36623134267dc6c520bd018ac5b8d59209ecd39bec294befec5667
May 28 15:10:49.226102 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.226082    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtvhp"
May 28 15:10:49.226176 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.226159    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 15:10:49.238737 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.238666    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtvhp"
May 28 15:10:49.289925 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.289884    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal" event={"ID":"b6642e2f3392820093d8ef926280a2cb","Type":"ContainerStarted","Data":"8802dd018f36623134267dc6c520bd018ac5b8d59209ecd39bec294befec5667"}
May 28 15:10:49.290739 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.290721    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal" event={"ID":"1691e4b38b2da49454ca8518fd1a04cb","Type":"ContainerStarted","Data":"f10e92e19e2238f921d85cb1b31e0e182b3750887ecb4ef99ed5e1a395a13d64"}
May 28 15:10:49.295044 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.295029    2582 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 28 15:10:49.302718 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.302700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-os-release\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.302718 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.302725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:49.302840 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.302744    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-run-netns\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.302840 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.302760    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-run-systemd\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.303109 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303080    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f11838e8-88b0-4035-85e4-99fd5e9026c3-hosts-file\") pod \"node-resolver-sd8jw\" (UID: \"f11838e8-88b0-4035-85e4-99fd5e9026c3\") " pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.303192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.302816    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-run-netns\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.303192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.303285 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-var-lib-cni-bin\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.303285 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303249    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-run-systemd\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.303285 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303265    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f11838e8-88b0-4035-85e4-99fd5e9026c3-hosts-file\") pod \"node-resolver-sd8jw\" (UID: \"f11838e8-88b0-4035-85e4-99fd5e9026c3\") " pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.303407 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303352    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-cni-netd\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.303453 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303405    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-sysconfig\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.303453 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-cni-dir\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.303534 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303523    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-sysconfig\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.303579 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303554    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sn8tg\" (UniqueName: \"kubernetes.io/projected/81941efe-61e4-45de-9f42-87037c076e64-kube-api-access-sn8tg\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.303627 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303600    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-socket-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.303669 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-cnibin\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.303713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303671    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qs8vm\" (UniqueName: \"kubernetes.io/projected/085514bb-7b92-43e7-b961-c3d99f6cd3a5-kube-api-access-qs8vm\") pod \"iptables-alerter-qd8qz\" (UID: \"085514bb-7b92-43e7-b961-c3d99f6cd3a5\") " pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.303713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303703    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-run-ovn-kubernetes\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.303792 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303734    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.303792 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303767    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-modprobe-d\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.303876 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303799    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-lib-modules\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.303876 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303830    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc64f60e-b2fb-46c2-b608-93ed763fbdc6-konnectivity-ca\") pod \"konnectivity-agent-9pvmp\" (UID: \"dc64f60e-b2fb-46c2-b608-93ed763fbdc6\") " pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:10:49.303876 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303856    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/085514bb-7b92-43e7-b961-c3d99f6cd3a5-iptables-alerter-script\") pod \"iptables-alerter-qd8qz\" (UID: \"085514bb-7b92-43e7-b961-c3d99f6cd3a5\") " pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.303989 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303895    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/085514bb-7b92-43e7-b961-c3d99f6cd3a5-host-slash\") pod \"iptables-alerter-qd8qz\" (UID: \"085514bb-7b92-43e7-b961-c3d99f6cd3a5\") " pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.303989 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303906    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-run-ovn-kubernetes\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.303989 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303562    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-cni-netd\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.303989 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303927    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-run-ovn\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.303989 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.303986    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-run-ovn\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.304217 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304034    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.304217 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304077    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d8b93da-b2de-4a1c-b264-604d7329745f-ovnkube-config\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.304217 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304112    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-var-lib-kubelet\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.304217 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304154    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-modprobe-d\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.304217 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304159    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81941efe-61e4-45de-9f42-87037c076e64-etc-tuned\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.304217 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304199    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8f4cz\" (UniqueName: \"kubernetes.io/projected/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-kube-api-access-8f4cz\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe05b8d-d2bb-40c3-9642-61b447da82ab-host\") pod \"node-ca-bjtv8\" (UID: \"2fe05b8d-d2bb-40c3-9642-61b447da82ab\") " pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-var-lib-kubelet\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304281    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-socket-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-hostroot\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304326    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc64f60e-b2fb-46c2-b608-93ed763fbdc6-agent-certs\") pod \"konnectivity-agent-9pvmp\" (UID: \"dc64f60e-b2fb-46c2-b608-93ed763fbdc6\") " pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304355    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-log-socket\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304387    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-systemd\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-cnibin\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqh7\" (UniqueName: \"kubernetes.io/projected/2f261580-fa49-4bdd-bffa-1dc49eb17669-kube-api-access-zrqh7\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.304467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304458    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgt4m\" (UniqueName: \"kubernetes.io/projected/60775ca8-b4ea-4176-bda3-58ad85bbc806-kube-api-access-dgt4m\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ccdd5a3-8460-4c56-863d-239d9c2da33f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzzr2\" (UniqueName: \"kubernetes.io/projected/2ccdd5a3-8460-4c56-863d-239d9c2da33f-kube-api-access-tzzr2\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304593    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-sysctl-conf\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304631    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304662    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304679    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-var-lib-kubelet\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-socket-dir-parent\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304754    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-log-socket\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.304875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.304803    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/085514bb-7b92-43e7-b961-c3d99f6cd3a5-host-slash\") pod \"iptables-alerter-qd8qz\" (UID: \"085514bb-7b92-43e7-b961-c3d99f6cd3a5\") " pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.305333 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305133    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d8b93da-b2de-4a1c-b264-604d7329745f-ovnkube-config\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.305333 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305306    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-sysctl-conf\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.305418 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305356    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/085514bb-7b92-43e7-b961-c3d99f6cd3a5-iptables-alerter-script\") pod \"iptables-alerter-qd8qz\" (UID: \"085514bb-7b92-43e7-b961-c3d99f6cd3a5\") " pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.305418 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305369    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-systemd\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.305500 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305423    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.305500 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-daemon-config\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.305500 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305491    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-run-openvswitch\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.305626 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305524    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d8b93da-b2de-4a1c-b264-604d7329745f-env-overrides\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.305626 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305556    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-kubernetes\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.305626 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ccdd5a3-8460-4c56-863d-239d9c2da33f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.305626 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fe05b8d-d2bb-40c3-9642-61b447da82ab-serviceca\") pod \"node-ca-bjtv8\" (UID: \"2fe05b8d-d2bb-40c3-9642-61b447da82ab\") " pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.305798 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-etc-kubernetes\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.305798 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305682    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d8b93da-b2de-4a1c-b264-604d7329745f-ovnkube-script-lib\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.305798 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305714    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-run\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.305798 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305744    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81941efe-61e4-45de-9f42-87037c076e64-tmp\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.305798 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305777    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-registration-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.306003 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f261580-fa49-4bdd-bffa-1dc49eb17669-cni-binary-copy\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.306003 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305857    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-run-netns\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.306003 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-conf-dir\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.306003 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305920    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-etc-openvswitch\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.306003 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305953    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-node-log\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.306003 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305975    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc64f60e-b2fb-46c2-b608-93ed763fbdc6-konnectivity-ca\") pod \"konnectivity-agent-9pvmp\" (UID: \"dc64f60e-b2fb-46c2-b608-93ed763fbdc6\") " pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:10:49.306003 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ccdd5a3-8460-4c56-863d-239d9c2da33f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.306308 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306044    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.306308 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.305524    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-lib-modules\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.306308 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306167    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-kubernetes\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.306308 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306226    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-registration-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.306308 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306277    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-run\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.306814 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306627    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d8b93da-b2de-4a1c-b264-604d7329745f-ovnkube-script-lib\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.306814 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306668    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d8b93da-b2de-4a1c-b264-604d7329745f-env-overrides\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.306814 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306694    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-etc-openvswitch\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.306814 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306745    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-node-log\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.307019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306807    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-run-openvswitch\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.307019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-var-lib-cni-multus\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.307019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.306965    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:49.307182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307016    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-kubelet\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.307182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307048    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-slash\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.307182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307051    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-kubelet\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.307182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307094    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d8b93da-b2de-4a1c-b264-604d7329745f-ovn-node-metrics-cert\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.307182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307108    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-slash\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.307182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307176    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-sys-fs\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.307433 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307286    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f11838e8-88b0-4035-85e4-99fd5e9026c3-tmp-dir\") pod \"node-resolver-sd8jw\" (UID: \"f11838e8-88b0-4035-85e4-99fd5e9026c3\") " pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.307433 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307370    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-sys-fs\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.307522 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307442    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-sysctl-d\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.307713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307694    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f11838e8-88b0-4035-85e4-99fd5e9026c3-tmp-dir\") pod \"node-resolver-sd8jw\" (UID: \"f11838e8-88b0-4035-85e4-99fd5e9026c3\") " pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.307867 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307841    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-etc-sysctl-d\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.307942 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307893    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-sys\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.308010 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-system-cni-dir\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.308010 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.307995    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74c6m\" (UniqueName: \"kubernetes.io/projected/f11838e8-88b0-4035-85e4-99fd5e9026c3-kube-api-access-74c6m\") pod \"node-resolver-sd8jw\" (UID: \"f11838e8-88b0-4035-85e4-99fd5e9026c3\") " pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.308142 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308052    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-var-lib-openvswitch\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.308142 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308110    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-sys\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.308243 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308191    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-cni-bin\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.308243 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308206    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-var-lib-openvswitch\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.308343 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308257    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-host-cni-bin\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.308343 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308311    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-host\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.308433 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308358    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81941efe-61e4-45de-9f42-87037c076e64-host\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.308433 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpzm7\" (UniqueName: \"kubernetes.io/projected/2fe05b8d-d2bb-40c3-9642-61b447da82ab-kube-api-access-vpzm7\") pod \"node-ca-bjtv8\" (UID: \"2fe05b8d-d2bb-40c3-9642-61b447da82ab\") " pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.308433 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-os-release\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.308576 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-run-k8s-cni-cncf-io\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.308576 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-run-multus-certs\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.308576 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308549    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-systemd-units\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.308718 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308582    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rfmlf\" (UniqueName: \"kubernetes.io/projected/3d8b93da-b2de-4a1c-b264-604d7329745f-kube-api-access-rfmlf\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.308718 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308615    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-device-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.308718 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308644    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d8b93da-b2de-4a1c-b264-604d7329745f-systemd-units\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.308718 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308649    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81941efe-61e4-45de-9f42-87037c076e64-etc-tuned\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.308718 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-system-cni-dir\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.308956 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308778    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-device-dir\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.308956 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308837    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81941efe-61e4-45de-9f42-87037c076e64-tmp\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.308956 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.308841    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc64f60e-b2fb-46c2-b608-93ed763fbdc6-agent-certs\") pod \"konnectivity-agent-9pvmp\" (UID: \"dc64f60e-b2fb-46c2-b608-93ed763fbdc6\") " pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:10:49.309539 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.309514    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d8b93da-b2de-4a1c-b264-604d7329745f-ovn-node-metrics-cert\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.310651 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.310630    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn8tg\" (UniqueName: \"kubernetes.io/projected/81941efe-61e4-45de-9f42-87037c076e64-kube-api-access-sn8tg\") pod \"tuned-dcgvr\" (UID: \"81941efe-61e4-45de-9f42-87037c076e64\") " pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.311306 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.311291    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f4cz\" (UniqueName: \"kubernetes.io/projected/95c19bfa-ba5d-4068-8cf3-1e76902be2a5-kube-api-access-8f4cz\") pod \"aws-ebs-csi-driver-node-hczrb\" (UID: \"95c19bfa-ba5d-4068-8cf3-1e76902be2a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.311474 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.311459    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs8vm\" (UniqueName: \"kubernetes.io/projected/085514bb-7b92-43e7-b961-c3d99f6cd3a5-kube-api-access-qs8vm\") pod \"iptables-alerter-qd8qz\" (UID: \"085514bb-7b92-43e7-b961-c3d99f6cd3a5\") " pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.312461 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.312439    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 15:10:49.312461 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.312457    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 15:10:49.312461 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.312466    2582 projected.go:194] Error preparing data for projected volume kube-api-access-hxqrq for pod openshift-network-diagnostics/network-check-target-j5kk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:49.312646 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.312510    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq podName:9884c711-a48a-419c-abc1-d50cf28d0b46 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:49.812496803 +0000 UTC m=+2.111566312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hxqrq" (UniqueName: "kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq") pod "network-check-target-j5kk5" (UID: "9884c711-a48a-419c-abc1-d50cf28d0b46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:49.314564 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.314548    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74c6m\" (UniqueName: \"kubernetes.io/projected/f11838e8-88b0-4035-85e4-99fd5e9026c3-kube-api-access-74c6m\") pod \"node-resolver-sd8jw\" (UID: \"f11838e8-88b0-4035-85e4-99fd5e9026c3\") " pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.316049 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.316031    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfmlf\" (UniqueName: \"kubernetes.io/projected/3d8b93da-b2de-4a1c-b264-604d7329745f-kube-api-access-rfmlf\") pod \"ovnkube-node-tx2n2\" (UID: \"3d8b93da-b2de-4a1c-b264-604d7329745f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.409355 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409327    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f261580-fa49-4bdd-bffa-1dc49eb17669-cni-binary-copy\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409355 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409358    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-run-netns\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409519 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409372    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-conf-dir\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409519 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409422    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-conf-dir\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409519 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409425    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ccdd5a3-8460-4c56-863d-239d9c2da33f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.409519 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409457    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-var-lib-cni-multus\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409519 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409501    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-system-cni-dir\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409686 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409563    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-system-cni-dir\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409686 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409574    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-var-lib-cni-multus\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409686 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409586    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vpzm7\" (UniqueName: \"kubernetes.io/projected/2fe05b8d-d2bb-40c3-9642-61b447da82ab-kube-api-access-vpzm7\") pod \"node-ca-bjtv8\" (UID: \"2fe05b8d-d2bb-40c3-9642-61b447da82ab\") " pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.409686 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409565    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-run-netns\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409686 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409614    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-os-release\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409686 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409637    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-run-k8s-cni-cncf-io\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409677    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-run-multus-certs\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409699    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-os-release\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409726    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-system-cni-dir\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409730    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-run-k8s-cni-cncf-io\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409750    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-os-release\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409742    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-run-multus-certs\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409773    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-system-cni-dir\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409774    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409822    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-os-release\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409821    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.409851    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409864    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-var-lib-cni-bin\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409895    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-cni-dir\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.409909    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs podName:60775ca8-b4ea-4176-bda3-58ad85bbc806 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:49.909891775 +0000 UTC m=+2.208961268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs") pod "network-metrics-daemon-9lnwm" (UID: "60775ca8-b4ea-4176-bda3-58ad85bbc806") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409935    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409943    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-cni-dir\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.409951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409949    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f261580-fa49-4bdd-bffa-1dc49eb17669-cni-binary-copy\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409986    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-cnibin\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409992    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ccdd5a3-8460-4c56-863d-239d9c2da33f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409987    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-var-lib-cni-bin\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.409938    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ccdd5a3-8460-4c56-863d-239d9c2da33f-cnibin\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410045    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe05b8d-d2bb-40c3-9642-61b447da82ab-host\") pod \"node-ca-bjtv8\" (UID: \"2fe05b8d-d2bb-40c3-9642-61b447da82ab\") " pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410062    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-var-lib-kubelet\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410092    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-hostroot\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410107    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-cnibin\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410153    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-host-var-lib-kubelet\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410161    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-hostroot\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410160    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-cnibin\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410176    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe05b8d-d2bb-40c3-9642-61b447da82ab-host\") pod \"node-ca-bjtv8\" (UID: \"2fe05b8d-d2bb-40c3-9642-61b447da82ab\") " pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410202    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqh7\" (UniqueName: \"kubernetes.io/projected/2f261580-fa49-4bdd-bffa-1dc49eb17669-kube-api-access-zrqh7\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410235    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dgt4m\" (UniqueName: \"kubernetes.io/projected/60775ca8-b4ea-4176-bda3-58ad85bbc806-kube-api-access-dgt4m\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410268    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ccdd5a3-8460-4c56-863d-239d9c2da33f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410295    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tzzr2\" (UniqueName: \"kubernetes.io/projected/2ccdd5a3-8460-4c56-863d-239d9c2da33f-kube-api-access-tzzr2\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410325    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-socket-dir-parent\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.410650 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410349    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-daemon-config\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.411108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410377    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ccdd5a3-8460-4c56-863d-239d9c2da33f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.411108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410394    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-socket-dir-parent\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.411108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410400    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fe05b8d-d2bb-40c3-9642-61b447da82ab-serviceca\") pod \"node-ca-bjtv8\" (UID: \"2fe05b8d-d2bb-40c3-9642-61b447da82ab\") " pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.411108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410425    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-etc-kubernetes\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.411108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410489    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f261580-fa49-4bdd-bffa-1dc49eb17669-etc-kubernetes\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.411108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410741    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ccdd5a3-8460-4c56-863d-239d9c2da33f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.411108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410765    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ccdd5a3-8460-4c56-863d-239d9c2da33f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.411108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410840    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fe05b8d-d2bb-40c3-9642-61b447da82ab-serviceca\") pod \"node-ca-bjtv8\" (UID: \"2fe05b8d-d2bb-40c3-9642-61b447da82ab\") " pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.411108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.410868    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f261580-fa49-4bdd-bffa-1dc49eb17669-multus-daemon-config\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.418366 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.418341    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpzm7\" (UniqueName: \"kubernetes.io/projected/2fe05b8d-d2bb-40c3-9642-61b447da82ab-kube-api-access-vpzm7\") pod \"node-ca-bjtv8\" (UID: \"2fe05b8d-d2bb-40c3-9642-61b447da82ab\") " pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.418742 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.418723    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqh7\" (UniqueName: \"kubernetes.io/projected/2f261580-fa49-4bdd-bffa-1dc49eb17669-kube-api-access-zrqh7\") pod \"multus-b9mt2\" (UID: \"2f261580-fa49-4bdd-bffa-1dc49eb17669\") " pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.418913 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.418898    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzzr2\" (UniqueName: \"kubernetes.io/projected/2ccdd5a3-8460-4c56-863d-239d9c2da33f-kube-api-access-tzzr2\") pod \"multus-additional-cni-plugins-lpq57\" (UID: \"2ccdd5a3-8460-4c56-863d-239d9c2da33f\") " pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.418993 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.418978    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgt4m\" (UniqueName: \"kubernetes.io/projected/60775ca8-b4ea-4176-bda3-58ad85bbc806-kube-api-access-dgt4m\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:49.509099 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.509022    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:10:49.515523 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.515500    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8b93da_b2de_4a1c_b264_604d7329745f.slice/crio-29769b689fa3b2d474ce0f062a685a176aa9532e1e114a49075a88cf7e9da914 WatchSource:0}: Error finding container 29769b689fa3b2d474ce0f062a685a176aa9532e1e114a49075a88cf7e9da914: Status 404 returned error can't find the container with id 29769b689fa3b2d474ce0f062a685a176aa9532e1e114a49075a88cf7e9da914
May 28 15:10:49.528716 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.528695    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:10:49.533897 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.533868    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc64f60e_b2fb_46c2_b608_93ed763fbdc6.slice/crio-db26280ab9c9d55a6e5ecd820387c2540930c39f0eb68f2e203bf3b8651dff79 WatchSource:0}: Error finding container db26280ab9c9d55a6e5ecd820387c2540930c39f0eb68f2e203bf3b8651dff79: Status 404 returned error can't find the container with id db26280ab9c9d55a6e5ecd820387c2540930c39f0eb68f2e203bf3b8651dff79
May 28 15:10:49.533996 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.533954    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dcgvr"
May 28 15:10:49.537774 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.537757    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sd8jw"
May 28 15:10:49.539986 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.539969    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81941efe_61e4_45de_9f42_87037c076e64.slice/crio-b2d5aab062c10ed1ee57541ba868fa21dcd13e43fef9437f65ff69674647c079 WatchSource:0}: Error finding container b2d5aab062c10ed1ee57541ba868fa21dcd13e43fef9437f65ff69674647c079: Status 404 returned error can't find the container with id b2d5aab062c10ed1ee57541ba868fa21dcd13e43fef9437f65ff69674647c079
May 28 15:10:49.544189 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.544167    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11838e8_88b0_4035_85e4_99fd5e9026c3.slice/crio-ee716d73d999f43930bb67e32bb6fad57fb2720e7b3e92121eb93092cd66b27f WatchSource:0}: Error finding container ee716d73d999f43930bb67e32bb6fad57fb2720e7b3e92121eb93092cd66b27f: Status 404 returned error can't find the container with id ee716d73d999f43930bb67e32bb6fad57fb2720e7b3e92121eb93092cd66b27f
May 28 15:10:49.544265 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.544205    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qd8qz"
May 28 15:10:49.549192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.549172    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb"
May 28 15:10:49.551412 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.551389    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085514bb_7b92_43e7_b961_c3d99f6cd3a5.slice/crio-46f58c80006813be835d5be19e574b4b26518c23c02cc620ddea3f21dba35702 WatchSource:0}: Error finding container 46f58c80006813be835d5be19e574b4b26518c23c02cc620ddea3f21dba35702: Status 404 returned error can't find the container with id 46f58c80006813be835d5be19e574b4b26518c23c02cc620ddea3f21dba35702
May 28 15:10:49.554192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.554173    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjtv8"
May 28 15:10:49.554925 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.554904    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c19bfa_ba5d_4068_8cf3_1e76902be2a5.slice/crio-8ba43e22986f6560263ba4fb604c3be87de187554159c6ae934078bbdace9c15 WatchSource:0}: Error finding container 8ba43e22986f6560263ba4fb604c3be87de187554159c6ae934078bbdace9c15: Status 404 returned error can't find the container with id 8ba43e22986f6560263ba4fb604c3be87de187554159c6ae934078bbdace9c15
May 28 15:10:49.559189 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.559161    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lpq57"
May 28 15:10:49.559678 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.559659    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe05b8d_d2bb_40c3_9642_61b447da82ab.slice/crio-4e8cc231557c5c4b261c9f3c1eb2672bf310be1a6f8994174d208c3a64df31d0 WatchSource:0}: Error finding container 4e8cc231557c5c4b261c9f3c1eb2672bf310be1a6f8994174d208c3a64df31d0: Status 404 returned error can't find the container with id 4e8cc231557c5c4b261c9f3c1eb2672bf310be1a6f8994174d208c3a64df31d0
May 28 15:10:49.563507 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.563490    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b9mt2"
May 28 15:10:49.564629 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.564611    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ccdd5a3_8460_4c56_863d_239d9c2da33f.slice/crio-36a7af0f2529acd91fc6e0406c4ae3a3a67c21c57e71bf481675bf7a190c1bd4 WatchSource:0}: Error finding container 36a7af0f2529acd91fc6e0406c4ae3a3a67c21c57e71bf481675bf7a190c1bd4: Status 404 returned error can't find the container with id 36a7af0f2529acd91fc6e0406c4ae3a3a67c21c57e71bf481675bf7a190c1bd4
May 28 15:10:49.569522 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:10:49.569503    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f261580_fa49_4bdd_bffa_1dc49eb17669.slice/crio-ce0b86e27d7c18d765b816a671b83ef40775aa84d73e7dd3e61ea554adc4ee3e WatchSource:0}: Error finding container ce0b86e27d7c18d765b816a671b83ef40775aa84d73e7dd3e61ea554adc4ee3e: Status 404 returned error can't find the container with id ce0b86e27d7c18d765b816a671b83ef40775aa84d73e7dd3e61ea554adc4ee3e
May 28 15:10:49.569613 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.569575    2582 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 15:10:49.813298 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.813193    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:49.813510 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.813489    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 15:10:49.813569 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.813524    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 15:10:49.813569 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.813540    2582 projected.go:194] Error preparing data for projected volume kube-api-access-hxqrq for pod openshift-network-diagnostics/network-check-target-j5kk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:49.813722 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.813708    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq podName:9884c711-a48a-419c-abc1-d50cf28d0b46 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:50.813666327 +0000 UTC m=+3.112735830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxqrq" (UniqueName: "kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq") pod "network-check-target-j5kk5" (UID: "9884c711-a48a-419c-abc1-d50cf28d0b46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:49.904861 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.904830    2582 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 15:10:49.914396 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:49.914369    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:49.914558 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.914540    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:49.914617 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:49.914607    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs podName:60775ca8-b4ea-4176-bda3-58ad85bbc806 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:50.914587212 +0000 UTC m=+3.213656716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs") pod "network-metrics-daemon-9lnwm" (UID: "60775ca8-b4ea-4176-bda3-58ad85bbc806") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:50.240211 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.240067    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 15:05:49 +0000 UTC" deadline="2027-12-10 13:43:24.853982104 +0000 UTC"
May 28 15:10:50.240211 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.240152    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13462h32m34.613835299s"
May 28 15:10:50.329494 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.329450    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b9mt2" event={"ID":"2f261580-fa49-4bdd-bffa-1dc49eb17669","Type":"ContainerStarted","Data":"ce0b86e27d7c18d765b816a671b83ef40775aa84d73e7dd3e61ea554adc4ee3e"}
May 28 15:10:50.342021 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.341924    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb" event={"ID":"95c19bfa-ba5d-4068-8cf3-1e76902be2a5","Type":"ContainerStarted","Data":"8ba43e22986f6560263ba4fb604c3be87de187554159c6ae934078bbdace9c15"}
May 28 15:10:50.347011 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.346962    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dcgvr" event={"ID":"81941efe-61e4-45de-9f42-87037c076e64","Type":"ContainerStarted","Data":"b2d5aab062c10ed1ee57541ba868fa21dcd13e43fef9437f65ff69674647c079"}
May 28 15:10:50.360893 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.360813    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9pvmp" event={"ID":"dc64f60e-b2fb-46c2-b608-93ed763fbdc6","Type":"ContainerStarted","Data":"db26280ab9c9d55a6e5ecd820387c2540930c39f0eb68f2e203bf3b8651dff79"}
May 28 15:10:50.384740 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.384528    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" event={"ID":"3d8b93da-b2de-4a1c-b264-604d7329745f","Type":"ContainerStarted","Data":"29769b689fa3b2d474ce0f062a685a176aa9532e1e114a49075a88cf7e9da914"}
May 28 15:10:50.393475 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.393449    2582 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 15:10:50.394933 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.394902    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpq57" event={"ID":"2ccdd5a3-8460-4c56-863d-239d9c2da33f","Type":"ContainerStarted","Data":"36a7af0f2529acd91fc6e0406c4ae3a3a67c21c57e71bf481675bf7a190c1bd4"}
May 28 15:10:50.406509 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.406477    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjtv8" event={"ID":"2fe05b8d-d2bb-40c3-9642-61b447da82ab","Type":"ContainerStarted","Data":"4e8cc231557c5c4b261c9f3c1eb2672bf310be1a6f8994174d208c3a64df31d0"}
May 28 15:10:50.417647 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.417609    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qd8qz" event={"ID":"085514bb-7b92-43e7-b961-c3d99f6cd3a5","Type":"ContainerStarted","Data":"46f58c80006813be835d5be19e574b4b26518c23c02cc620ddea3f21dba35702"}
May 28 15:10:50.428904 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.428850    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sd8jw" event={"ID":"f11838e8-88b0-4035-85e4-99fd5e9026c3","Type":"ContainerStarted","Data":"ee716d73d999f43930bb67e32bb6fad57fb2720e7b3e92121eb93092cd66b27f"}
May 28 15:10:50.822102 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.822061    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:50.822291 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:50.822239    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 15:10:50.822291 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:50.822261    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 15:10:50.822291 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:50.822273    2582 projected.go:194] Error preparing data for projected volume kube-api-access-hxqrq for pod openshift-network-diagnostics/network-check-target-j5kk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:50.822470 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:50.822332    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq podName:9884c711-a48a-419c-abc1-d50cf28d0b46 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:52.822314157 +0000 UTC m=+5.121383651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxqrq" (UniqueName: "kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq") pod "network-check-target-j5kk5" (UID: "9884c711-a48a-419c-abc1-d50cf28d0b46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:50.922933 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:50.922895    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:50.923157 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:50.923138    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:50.923238 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:50.923210    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs podName:60775ca8-b4ea-4176-bda3-58ad85bbc806 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:52.923190016 +0000 UTC m=+5.222259508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs") pod "network-metrics-daemon-9lnwm" (UID: "60775ca8-b4ea-4176-bda3-58ad85bbc806") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:51.240911 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:51.240805    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 15:05:49 +0000 UTC" deadline="2027-12-06 21:56:23.330579043 +0000 UTC"
May 28 15:10:51.240911 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:51.240854    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13374h45m32.08972884s"
May 28 15:10:51.288955 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:51.288216    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:51.288955 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:51.288358    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:10:51.288955 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:51.288798    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:51.288955 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:51.288888    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:10:52.043397 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:52.043360    2582 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 15:10:52.838599 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:52.838557    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:52.839106 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:52.838725    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 15:10:52.839106 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:52.838745    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 15:10:52.839106 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:52.838757    2582 projected.go:194] Error preparing data for projected volume kube-api-access-hxqrq for pod openshift-network-diagnostics/network-check-target-j5kk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:52.839106 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:52.838818    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq podName:9884c711-a48a-419c-abc1-d50cf28d0b46 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:56.838797741 +0000 UTC m=+9.137867230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxqrq" (UniqueName: "kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq") pod "network-check-target-j5kk5" (UID: "9884c711-a48a-419c-abc1-d50cf28d0b46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:52.939455 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:52.939417    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:52.939630 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:52.939570    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:52.939687 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:52.939636    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs podName:60775ca8-b4ea-4176-bda3-58ad85bbc806 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:56.939617703 +0000 UTC m=+9.238687206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs") pod "network-metrics-daemon-9lnwm" (UID: "60775ca8-b4ea-4176-bda3-58ad85bbc806") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:53.287835 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:53.287797    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:53.287988 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:53.287846    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:53.287988 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:53.287944    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:10:53.288095 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:53.288034    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:10:55.287630 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:55.287596    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:55.288076 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:55.287596    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:55.288076 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:55.287740    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:10:55.288076 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:55.287817    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:10:56.874286 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:56.874077    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:56.874717 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:56.874296    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 15:10:56.874717 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:56.874313    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 15:10:56.874717 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:56.874323    2582 projected.go:194] Error preparing data for projected volume kube-api-access-hxqrq for pod openshift-network-diagnostics/network-check-target-j5kk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:56.874717 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:56.874391    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq podName:9884c711-a48a-419c-abc1-d50cf28d0b46 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:04.874372689 +0000 UTC m=+17.173442190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxqrq" (UniqueName: "kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq") pod "network-check-target-j5kk5" (UID: "9884c711-a48a-419c-abc1-d50cf28d0b46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:10:56.975446 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:56.975345    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:56.975616 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:56.975517    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:56.975616 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:56.975597    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs podName:60775ca8-b4ea-4176-bda3-58ad85bbc806 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:04.975572776 +0000 UTC m=+17.274642301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs") pod "network-metrics-daemon-9lnwm" (UID: "60775ca8-b4ea-4176-bda3-58ad85bbc806") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:10:57.288386 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:57.288292    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:57.288386 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:57.288326    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:57.288672 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:57.288438    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:10:57.288875 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:57.288839    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:10:59.287536 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:59.287507    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:10:59.287943 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:59.287621    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:10:59.287943 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:10:59.287508    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:10:59.287943 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:10:59.287814    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:01.288240 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:01.288198    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:01.288678 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:01.288198    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:01.288678 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:01.288326    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:01.288678 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:01.288435    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:03.287939 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:03.287906    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:03.288445 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:03.287916    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:03.288445 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:03.288025    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:03.288445 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:03.288135    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:04.932309 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:04.932272    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:04.932701 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:04.932427    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 15:11:04.932701 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:04.932453    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 15:11:04.932701 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:04.932467    2582 projected.go:194] Error preparing data for projected volume kube-api-access-hxqrq for pod openshift-network-diagnostics/network-check-target-j5kk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:11:04.932701 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:04.932521    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq podName:9884c711-a48a-419c-abc1-d50cf28d0b46 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:20.932508001 +0000 UTC m=+33.231577491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxqrq" (UniqueName: "kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq") pod "network-check-target-j5kk5" (UID: "9884c711-a48a-419c-abc1-d50cf28d0b46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:11:05.033440 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:05.033400    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:05.033603 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:05.033538    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:11:05.033656 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:05.033609    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs podName:60775ca8-b4ea-4176-bda3-58ad85bbc806 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:21.033589634 +0000 UTC m=+33.332659139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs") pod "network-metrics-daemon-9lnwm" (UID: "60775ca8-b4ea-4176-bda3-58ad85bbc806") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:11:05.287985 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:05.287903    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:05.288151 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:05.288036    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:05.288151 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:05.288099    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:05.288261 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:05.288218    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:07.288517 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:07.288480    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:07.288880 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:07.288476    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:07.288880 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:07.288592    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:07.288880 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:07.288653    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:08.463710 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.463323    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sd8jw" event={"ID":"f11838e8-88b0-4035-85e4-99fd5e9026c3","Type":"ContainerStarted","Data":"0bb0300b5c93e5fe664b3e5619c19a2b761c090232ec5f2d9aa24f40af53a83f"}
May 28 15:11:08.464573 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.464553    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b9mt2" event={"ID":"2f261580-fa49-4bdd-bffa-1dc49eb17669","Type":"ContainerStarted","Data":"31177360072cbd947f7606a9b51cc4b45b2890f88c2f05333ed39a7db631a3ae"}
May 28 15:11:08.465836 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.465815    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb" event={"ID":"95c19bfa-ba5d-4068-8cf3-1e76902be2a5","Type":"ContainerStarted","Data":"2d9bf401295385be37a7e7d589147926df5c2ffa93d5dda1d079552c2ebffe0b"}
May 28 15:11:08.467319 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.467294    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dcgvr" event={"ID":"81941efe-61e4-45de-9f42-87037c076e64","Type":"ContainerStarted","Data":"d29e2b99eba2f171852dc76ebc8970d4a06e6e6e358f5a1f5cd724dc664e3ab5"}
May 28 15:11:08.468653 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.468631    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9pvmp" event={"ID":"dc64f60e-b2fb-46c2-b608-93ed763fbdc6","Type":"ContainerStarted","Data":"d72586adf08be8faf0414dbbbdf6ea267d9a97d978877067a6681852eda16506"}
May 28 15:11:08.471131 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.471096    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" event={"ID":"3d8b93da-b2de-4a1c-b264-604d7329745f","Type":"ContainerStarted","Data":"044b2d61a80574b002c24eaffa7c03f10daa65a22467a2e89930f576b2e12811"}
May 28 15:11:08.471218 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.471140    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" event={"ID":"3d8b93da-b2de-4a1c-b264-604d7329745f","Type":"ContainerStarted","Data":"d8f8c7570c8e5610b29da913c8e4df49719b57d810baf4fd6ffe890c8f9b1859"}
May 28 15:11:08.471218 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.471154    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" event={"ID":"3d8b93da-b2de-4a1c-b264-604d7329745f","Type":"ContainerStarted","Data":"8795e23d141aa2c74dc7873bd59fcaa84125dd7057d602ebd28df8d691d7ce6c"}
May 28 15:11:08.471218 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.471166    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" event={"ID":"3d8b93da-b2de-4a1c-b264-604d7329745f","Type":"ContainerStarted","Data":"a0d583a7533def42003c09c2360e86b0a32a56b14ec5f3eed306a2535352ff27"}
May 28 15:11:08.471218 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.471178    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" event={"ID":"3d8b93da-b2de-4a1c-b264-604d7329745f","Type":"ContainerStarted","Data":"6035c2606122a4a9c4a0bae2e0381e066fabaab523c655607bae618db6c945d3"}
May 28 15:11:08.471218 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.471213    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" event={"ID":"3d8b93da-b2de-4a1c-b264-604d7329745f","Type":"ContainerStarted","Data":"6caceb2685615806e70d5b9278fed73f11f1d942a672564351d0a562d8ef12a8"}
May 28 15:11:08.472893 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.472871    2582 generic.go:358] "Generic (PLEG): container finished" podID="b6642e2f3392820093d8ef926280a2cb" containerID="7414f0543d1f648c5fb99dc22059672e18eaa071e71b8549d6f8008bde0430fa" exitCode=0
May 28 15:11:08.472969 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.472937    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal" event={"ID":"b6642e2f3392820093d8ef926280a2cb","Type":"ContainerDied","Data":"7414f0543d1f648c5fb99dc22059672e18eaa071e71b8549d6f8008bde0430fa"}
May 28 15:11:08.474255 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.474236    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal" event={"ID":"1691e4b38b2da49454ca8518fd1a04cb","Type":"ContainerStarted","Data":"184688b1cbed17a5a1e23ab9836a4ef8968810bd377099774165064afc3fb634"}
May 28 15:11:08.475696 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.475674    2582 generic.go:358] "Generic (PLEG): container finished" podID="2ccdd5a3-8460-4c56-863d-239d9c2da33f" containerID="5931b0e892539fa29dccc04504ae95188dbf612c9ccd7e3b08f9912ee4d85062" exitCode=0
May 28 15:11:08.475806 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.475742    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpq57" event={"ID":"2ccdd5a3-8460-4c56-863d-239d9c2da33f","Type":"ContainerDied","Data":"5931b0e892539fa29dccc04504ae95188dbf612c9ccd7e3b08f9912ee4d85062"}
May 28 15:11:08.477372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.477348    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjtv8" event={"ID":"2fe05b8d-d2bb-40c3-9642-61b447da82ab","Type":"ContainerStarted","Data":"a8d66c96d629240ea5341730f311ff7e919e207029e1159eb0ef2951fde5d0c7"}
May 28 15:11:08.481218 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.481168    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-sd8jw" podStartSLOduration=2.367614083 podStartE2EDuration="20.481153601s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:10:49.545574227 +0000 UTC m=+1.844643716" lastFinishedPulling="2026-05-28 15:11:07.659113729 +0000 UTC m=+19.958183234" observedRunningTime="2026-05-28 15:11:08.480291472 +0000 UTC m=+20.779360995" watchObservedRunningTime="2026-05-28 15:11:08.481153601 +0000 UTC m=+20.780223113"
May 28 15:11:08.511240 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.510972    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-203.ec2.internal" podStartSLOduration=19.510958796 podStartE2EDuration="19.510958796s" podCreationTimestamp="2026-05-28 15:10:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:11:08.51040547 +0000 UTC m=+20.809474982" watchObservedRunningTime="2026-05-28 15:11:08.510958796 +0000 UTC m=+20.810028307"
May 28 15:11:08.540137 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.540071    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bjtv8" podStartSLOduration=2.47000804 podStartE2EDuration="20.540055082s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:10:49.561867897 +0000 UTC m=+1.860937400" lastFinishedPulling="2026-05-28 15:11:07.631914949 +0000 UTC m=+19.930984442" observedRunningTime="2026-05-28 15:11:08.539973295 +0000 UTC m=+20.839042809" watchObservedRunningTime="2026-05-28 15:11:08.540055082 +0000 UTC m=+20.839124594"
May 28 15:11:08.540471 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.540436    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9pvmp" podStartSLOduration=2.8634570889999997 podStartE2EDuration="20.540427408s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:10:49.535386368 +0000 UTC m=+1.834455858" lastFinishedPulling="2026-05-28 15:11:07.212356676 +0000 UTC m=+19.511426177" observedRunningTime="2026-05-28 15:11:08.525214175 +0000 UTC m=+20.824283688" watchObservedRunningTime="2026-05-28 15:11:08.540427408 +0000 UTC m=+20.839496916"
May 28 15:11:08.595288 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.595192    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-b9mt2" podStartSLOduration=2.438119819 podStartE2EDuration="20.59517419s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:10:49.570845351 +0000 UTC m=+1.869914843" lastFinishedPulling="2026-05-28 15:11:07.727899711 +0000 UTC m=+20.026969214" observedRunningTime="2026-05-28 15:11:08.594874089 +0000 UTC m=+20.893943603" watchObservedRunningTime="2026-05-28 15:11:08.59517419 +0000 UTC m=+20.894243702"
May 28 15:11:08.595530 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.595501    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dcgvr" podStartSLOduration=2.477933675 podStartE2EDuration="20.595492755s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:10:49.541771509 +0000 UTC m=+1.840840998" lastFinishedPulling="2026-05-28 15:11:07.659330586 +0000 UTC m=+19.958400078" observedRunningTime="2026-05-28 15:11:08.57756789 +0000 UTC m=+20.876637400" watchObservedRunningTime="2026-05-28 15:11:08.595492755 +0000 UTC m=+20.894562267"
May 28 15:11:08.963463 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:08.963436    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 28 15:11:09.259886 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:09.259726    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-28T15:11:08.963459607Z","UUID":"c87b07be-27e4-45ff-8d17-d798857d2b0d","Handler":null,"Name":"","Endpoint":""}
May 28 15:11:09.261403 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:09.261380    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 28 15:11:09.261532 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:09.261411    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 28 15:11:09.288211 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:09.288184    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:09.288211 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:09.288218    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:09.288390 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:09.288281    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:09.288390 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:09.288342    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:09.482005 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:09.481951    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal" event={"ID":"b6642e2f3392820093d8ef926280a2cb","Type":"ContainerStarted","Data":"290699d09633faaa3e0d303660b3f091132e64060a527ded96e89748c6c20d54"}
May 28 15:11:09.483952 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:09.483916    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qd8qz" event={"ID":"085514bb-7b92-43e7-b961-c3d99f6cd3a5","Type":"ContainerStarted","Data":"863e65a5a69ff9f7c6c5b1f9918e639aad14587dc606c51a1a95a4ff9dbdc81b"}
May 28 15:11:09.485820 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:09.485787    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb" event={"ID":"95c19bfa-ba5d-4068-8cf3-1e76902be2a5","Type":"ContainerStarted","Data":"5e29a17a4ac46085e9ffa8aab21505c81d9eaa91a75def09fd161de1f8f5b9e2"}
May 28 15:11:09.497935 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:09.497878    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-203.ec2.internal" podStartSLOduration=20.497863482 podStartE2EDuration="20.497863482s" podCreationTimestamp="2026-05-28 15:10:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:11:09.497094841 +0000 UTC m=+21.796164351" watchObservedRunningTime="2026-05-28 15:11:09.497863482 +0000 UTC m=+21.796932994"
May 28 15:11:10.378387 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:10.378333    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:11:10.379146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:10.379109    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:11:10.393253 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:10.393209    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qd8qz" podStartSLOduration=4.365423822 podStartE2EDuration="22.393192524s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:10:49.55283773 +0000 UTC m=+1.851907222" lastFinishedPulling="2026-05-28 15:11:07.580606419 +0000 UTC m=+19.879675924" observedRunningTime="2026-05-28 15:11:09.509956088 +0000 UTC m=+21.809025599" watchObservedRunningTime="2026-05-28 15:11:10.393192524 +0000 UTC m=+22.692262037"
May 28 15:11:10.489877 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:10.489842    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb" event={"ID":"95c19bfa-ba5d-4068-8cf3-1e76902be2a5","Type":"ContainerStarted","Data":"529607b3d3e1466dd556f57c3db5d5362e850537410d682c6078dbbad206b4b0"}
May 28 15:11:10.493398 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:10.493359    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" event={"ID":"3d8b93da-b2de-4a1c-b264-604d7329745f","Type":"ContainerStarted","Data":"b6e9257c52bdaa99b11469bc133a7fd7441f3038b026fecbbf85c3e9ba2adb27"}
May 28 15:11:10.493965 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:10.493944    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:11:10.494785 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:10.494758    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9pvmp"
May 28 15:11:10.507179 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:10.507116    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hczrb" podStartSLOduration=2.278287927 podStartE2EDuration="22.507103816s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:10:49.5567438 +0000 UTC m=+1.855813289" lastFinishedPulling="2026-05-28 15:11:09.785559687 +0000 UTC m=+22.084629178" observedRunningTime="2026-05-28 15:11:10.5069658 +0000 UTC m=+22.806035310" watchObservedRunningTime="2026-05-28 15:11:10.507103816 +0000 UTC m=+22.806173327"
May 28 15:11:11.287564 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:11.287528    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:11.287749 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:11.287528    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:11.287749 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:11.287645    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:11.287874 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:11.287739    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:13.287964 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:13.287773    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:13.288427 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:13.287773    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:13.288427 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:13.288066    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:13.288427 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:13.288134    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:14.502416 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:14.502238    2582 generic.go:358] "Generic (PLEG): container finished" podID="2ccdd5a3-8460-4c56-863d-239d9c2da33f" containerID="4327f5fc9a60179c4481e27f9fefa6b52eb800addb1b7a6ac4c91ab999943a70" exitCode=0
May 28 15:11:14.502934 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:14.502315    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpq57" event={"ID":"2ccdd5a3-8460-4c56-863d-239d9c2da33f","Type":"ContainerDied","Data":"4327f5fc9a60179c4481e27f9fefa6b52eb800addb1b7a6ac4c91ab999943a70"}
May 28 15:11:14.505603 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:14.505583    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" event={"ID":"3d8b93da-b2de-4a1c-b264-604d7329745f","Type":"ContainerStarted","Data":"721477c6a721137cb3b9749947b2ea2bf60a5daf4e739e4cba43fde4037599db"}
May 28 15:11:14.505871 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:14.505859    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:11:14.505940 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:14.505876    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:11:14.520769 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:14.520746    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:11:14.550977 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:14.550936    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2" podStartSLOduration=8.357239619 podStartE2EDuration="26.550924149s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:10:49.516967719 +0000 UTC m=+1.816037209" lastFinishedPulling="2026-05-28 15:11:07.710652238 +0000 UTC m=+20.009721739" observedRunningTime="2026-05-28 15:11:14.549759083 +0000 UTC m=+26.848828595" watchObservedRunningTime="2026-05-28 15:11:14.550924149 +0000 UTC m=+26.849993659"
May 28 15:11:15.287883 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.287854    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:15.288053 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.287887    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:15.288053 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:15.288002    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:15.288455 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:15.288148    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:15.488883 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.488843    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5kk5"]
May 28 15:11:15.491755 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.491731    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9lnwm"]
May 28 15:11:15.509375 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.509344    2582 generic.go:358] "Generic (PLEG): container finished" podID="2ccdd5a3-8460-4c56-863d-239d9c2da33f" containerID="002bcf95af1548a41c5811a57d3221706ce7c1ca3fb5056311e0a92e2a381f8c" exitCode=0
May 28 15:11:15.510211 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.509425    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:15.510211 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.509435    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpq57" event={"ID":"2ccdd5a3-8460-4c56-863d-239d9c2da33f","Type":"ContainerDied","Data":"002bcf95af1548a41c5811a57d3221706ce7c1ca3fb5056311e0a92e2a381f8c"}
May 28 15:11:15.510211 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:15.509554    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:15.510211 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.509710    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:15.510211 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:15.509792    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:15.510808 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.510245    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:11:15.526885 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:15.526838    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:11:16.512907 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:16.512816    2582 generic.go:358] "Generic (PLEG): container finished" podID="2ccdd5a3-8460-4c56-863d-239d9c2da33f" containerID="6d323193f30586406e79a992d5bdbff485a24dc6e6161ede6204438358ccd057" exitCode=0
May 28 15:11:16.513291 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:16.512897    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpq57" event={"ID":"2ccdd5a3-8460-4c56-863d-239d9c2da33f","Type":"ContainerDied","Data":"6d323193f30586406e79a992d5bdbff485a24dc6e6161ede6204438358ccd057"}
May 28 15:11:17.287879 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:17.287799    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:17.288041 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:17.287799    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:17.288041 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:17.287934    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:17.288041 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:17.288020    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:19.288051 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:19.287807    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:19.288603 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:19.287889    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:19.288603 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:19.288169    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5kk5" podUID="9884c711-a48a-419c-abc1-d50cf28d0b46"
May 28 15:11:19.288603 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:19.288263    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:11:20.516820 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.516751    2582 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-203.ec2.internal" event="NodeReady"
May 28 15:11:20.517250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.516908    2582 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 28 15:11:20.557442 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.557406    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9n5pb"]
May 28 15:11:20.577779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.577748    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5lwgs"]
May 28 15:11:20.577961 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.577941    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.580786 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.580752    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 28 15:11:20.583017 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.581554    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 28 15:11:20.583280 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.583227    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8f6nh\""
May 28 15:11:20.591272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.591254    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9n5pb"]
May 28 15:11:20.591378 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.591280    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5lwgs"]
May 28 15:11:20.591434 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.591379    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:20.593776 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.593756    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 28 15:11:20.593876 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.593819    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 28 15:11:20.593876 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.593819    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 28 15:11:20.593993 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.593925    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ljmrm\""
May 28 15:11:20.647996 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.647960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713bf691-c64b-485f-b8f8-7094daac71eb-tmp-dir\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.648188 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.648007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfgg4\" (UniqueName: \"kubernetes.io/projected/713bf691-c64b-485f-b8f8-7094daac71eb-kube-api-access-hfgg4\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.648188 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.648103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/713bf691-c64b-485f-b8f8-7094daac71eb-config-volume\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.648301 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.648187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.749453 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.749413    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/713bf691-c64b-485f-b8f8-7094daac71eb-config-volume\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.749627 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.749483    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.749627 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.749507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:20.749627 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.749527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt9d8\" (UniqueName: \"kubernetes.io/projected/37fbb52e-9da0-488d-b0a2-8706e14120a8-kube-api-access-pt9d8\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:20.749627 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.749570    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713bf691-c64b-485f-b8f8-7094daac71eb-tmp-dir\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.749627 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.749588    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hfgg4\" (UniqueName: \"kubernetes.io/projected/713bf691-c64b-485f-b8f8-7094daac71eb-kube-api-access-hfgg4\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.749872 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:20.749648    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 15:11:20.749872 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:20.749736    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls podName:713bf691-c64b-485f-b8f8-7094daac71eb nodeName:}" failed. No retries permitted until 2026-05-28 15:11:21.249714704 +0000 UTC m=+33.548784197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls") pod "dns-default-9n5pb" (UID: "713bf691-c64b-485f-b8f8-7094daac71eb") : secret "dns-default-metrics-tls" not found
May 28 15:11:20.749989 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.749887    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713bf691-c64b-485f-b8f8-7094daac71eb-tmp-dir\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.750092 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.750074    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/713bf691-c64b-485f-b8f8-7094daac71eb-config-volume\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.760365 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.760344    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfgg4\" (UniqueName: \"kubernetes.io/projected/713bf691-c64b-485f-b8f8-7094daac71eb-kube-api-access-hfgg4\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:20.850469 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.850383    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:20.850469 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.850428    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pt9d8\" (UniqueName: \"kubernetes.io/projected/37fbb52e-9da0-488d-b0a2-8706e14120a8-kube-api-access-pt9d8\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:20.850748 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:20.850542    2582 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 15:11:20.850748 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:20.850632    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert podName:37fbb52e-9da0-488d-b0a2-8706e14120a8 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:21.350610421 +0000 UTC m=+33.649679913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert") pod "ingress-canary-5lwgs" (UID: "37fbb52e-9da0-488d-b0a2-8706e14120a8") : secret "canary-serving-cert" not found
May 28 15:11:20.859812 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.859785    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt9d8\" (UniqueName: \"kubernetes.io/projected/37fbb52e-9da0-488d-b0a2-8706e14120a8-kube-api-access-pt9d8\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:20.951149 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:20.951095    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:20.951343 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:20.951281    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 15:11:20.951343 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:20.951308    2582 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 15:11:20.951343 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:20.951321    2582 projected.go:194] Error preparing data for projected volume kube-api-access-hxqrq for pod openshift-network-diagnostics/network-check-target-j5kk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:11:20.951477 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:20.951384    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq podName:9884c711-a48a-419c-abc1-d50cf28d0b46 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:52.951365346 +0000 UTC m=+65.250434840 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxqrq" (UniqueName: "kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq") pod "network-check-target-j5kk5" (UID: "9884c711-a48a-419c-abc1-d50cf28d0b46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 15:11:21.052457 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.052414    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:21.052640 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:21.052551    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:11:21.052689 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:21.052659    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs podName:60775ca8-b4ea-4176-bda3-58ad85bbc806 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:53.052642611 +0000 UTC m=+65.351712101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs") pod "network-metrics-daemon-9lnwm" (UID: "60775ca8-b4ea-4176-bda3-58ad85bbc806") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 15:11:21.254158 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.254104    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:21.254376 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:21.254275    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 15:11:21.254376 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:21.254342    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls podName:713bf691-c64b-485f-b8f8-7094daac71eb nodeName:}" failed. No retries permitted until 2026-05-28 15:11:22.254328267 +0000 UTC m=+34.553397757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls") pod "dns-default-9n5pb" (UID: "713bf691-c64b-485f-b8f8-7094daac71eb") : secret "dns-default-metrics-tls" not found
May 28 15:11:21.287611 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.287572    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:21.287611 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.287619    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:21.290321 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.290295    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 15:11:21.290446 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.290300    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 15:11:21.290592 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.290573    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-49m45\""
May 28 15:11:21.290708 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.290685    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwwqm\""
May 28 15:11:21.290829 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.290779    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 15:11:21.354638 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:21.354601    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:21.354829 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:21.354739    2582 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 15:11:21.354829 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:21.354804    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert podName:37fbb52e-9da0-488d-b0a2-8706e14120a8 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:22.354785674 +0000 UTC m=+34.653855167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert") pod "ingress-canary-5lwgs" (UID: "37fbb52e-9da0-488d-b0a2-8706e14120a8") : secret "canary-serving-cert" not found
May 28 15:11:22.261107 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:22.261070    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:22.261713 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:22.261259    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 15:11:22.261713 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:22.261339    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls podName:713bf691-c64b-485f-b8f8-7094daac71eb nodeName:}" failed. No retries permitted until 2026-05-28 15:11:24.261315927 +0000 UTC m=+36.560385438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls") pod "dns-default-9n5pb" (UID: "713bf691-c64b-485f-b8f8-7094daac71eb") : secret "dns-default-metrics-tls" not found
May 28 15:11:22.362229 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:22.362189    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:22.362409 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:22.362323    2582 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 15:11:22.362409 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:22.362400    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert podName:37fbb52e-9da0-488d-b0a2-8706e14120a8 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:24.362379095 +0000 UTC m=+36.661448589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert") pod "ingress-canary-5lwgs" (UID: "37fbb52e-9da0-488d-b0a2-8706e14120a8") : secret "canary-serving-cert" not found
May 28 15:11:23.528399 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:23.528328    2582 generic.go:358] "Generic (PLEG): container finished" podID="2ccdd5a3-8460-4c56-863d-239d9c2da33f" containerID="924e3e23d7468b8bf8fe6485a80d1a358e06e53eff7e9662529c3f000cfc5f7c" exitCode=0
May 28 15:11:23.528399 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:23.528380    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpq57" event={"ID":"2ccdd5a3-8460-4c56-863d-239d9c2da33f","Type":"ContainerDied","Data":"924e3e23d7468b8bf8fe6485a80d1a358e06e53eff7e9662529c3f000cfc5f7c"}
May 28 15:11:24.275425 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:24.275242    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:24.275583 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:24.275387    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 15:11:24.275583 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:24.275516    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls podName:713bf691-c64b-485f-b8f8-7094daac71eb nodeName:}" failed. No retries permitted until 2026-05-28 15:11:28.275502429 +0000 UTC m=+40.574571918 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls") pod "dns-default-9n5pb" (UID: "713bf691-c64b-485f-b8f8-7094daac71eb") : secret "dns-default-metrics-tls" not found
May 28 15:11:24.376113 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:24.376072    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:24.376290 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:24.376207    2582 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 15:11:24.376290 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:24.376265    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert podName:37fbb52e-9da0-488d-b0a2-8706e14120a8 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:28.376250926 +0000 UTC m=+40.675320415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert") pod "ingress-canary-5lwgs" (UID: "37fbb52e-9da0-488d-b0a2-8706e14120a8") : secret "canary-serving-cert" not found
May 28 15:11:24.533060 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:24.532982    2582 generic.go:358] "Generic (PLEG): container finished" podID="2ccdd5a3-8460-4c56-863d-239d9c2da33f" containerID="92514f919ae62f297c65f8f927abc662499f7a478933876fe2889baa496cc2c3" exitCode=0
May 28 15:11:24.533060 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:24.533028    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpq57" event={"ID":"2ccdd5a3-8460-4c56-863d-239d9c2da33f","Type":"ContainerDied","Data":"92514f919ae62f297c65f8f927abc662499f7a478933876fe2889baa496cc2c3"}
May 28 15:11:25.537318 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:25.537278    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpq57" event={"ID":"2ccdd5a3-8460-4c56-863d-239d9c2da33f","Type":"ContainerStarted","Data":"efa855c533fbd2554aa1b712504ae7557f9b566861511ee18045ca84f15ed469"}
May 28 15:11:25.559009 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:25.558970    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lpq57" podStartSLOduration=4.137924675 podStartE2EDuration="37.558948103s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:10:49.565930099 +0000 UTC m=+1.864999588" lastFinishedPulling="2026-05-28 15:11:22.986953523 +0000 UTC m=+35.286023016" observedRunningTime="2026-05-28 15:11:25.558039963 +0000 UTC m=+37.857109473" watchObservedRunningTime="2026-05-28 15:11:25.558948103 +0000 UTC m=+37.858017614"
May 28 15:11:28.302139 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:28.302097    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:28.302678 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:28.302233    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 15:11:28.302678 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:28.302308    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls podName:713bf691-c64b-485f-b8f8-7094daac71eb nodeName:}" failed. No retries permitted until 2026-05-28 15:11:36.302286181 +0000 UTC m=+48.601355684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls") pod "dns-default-9n5pb" (UID: "713bf691-c64b-485f-b8f8-7094daac71eb") : secret "dns-default-metrics-tls" not found
May 28 15:11:28.403443 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:28.403405    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:28.403580 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:28.403523    2582 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 15:11:28.403580 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:28.403576    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert podName:37fbb52e-9da0-488d-b0a2-8706e14120a8 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:36.403559746 +0000 UTC m=+48.702629236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert") pod "ingress-canary-5lwgs" (UID: "37fbb52e-9da0-488d-b0a2-8706e14120a8") : secret "canary-serving-cert" not found
May 28 15:11:36.361256 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:36.361218    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:36.361614 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:36.361370    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 15:11:36.361614 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:36.361437    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls podName:713bf691-c64b-485f-b8f8-7094daac71eb nodeName:}" failed. No retries permitted until 2026-05-28 15:11:52.361419497 +0000 UTC m=+64.660488987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls") pod "dns-default-9n5pb" (UID: "713bf691-c64b-485f-b8f8-7094daac71eb") : secret "dns-default-metrics-tls" not found
May 28 15:11:36.461893 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:36.461854    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:36.462057 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:36.462012    2582 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 15:11:36.462102 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:36.462084    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert podName:37fbb52e-9da0-488d-b0a2-8706e14120a8 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:52.462067671 +0000 UTC m=+64.761137165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert") pod "ingress-canary-5lwgs" (UID: "37fbb52e-9da0-488d-b0a2-8706e14120a8") : secret "canary-serving-cert" not found
May 28 15:11:47.526421 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:47.526390    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tx2n2"
May 28 15:11:52.369736 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:52.369699    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:11:52.370143 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:52.369802    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 15:11:52.370143 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:52.369858    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls podName:713bf691-c64b-485f-b8f8-7094daac71eb nodeName:}" failed. No retries permitted until 2026-05-28 15:12:24.369844707 +0000 UTC m=+96.668914196 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls") pod "dns-default-9n5pb" (UID: "713bf691-c64b-485f-b8f8-7094daac71eb") : secret "dns-default-metrics-tls" not found
May 28 15:11:52.470087 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:52.470048    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:11:52.470272 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:52.470207    2582 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 15:11:52.470321 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:52.470278    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert podName:37fbb52e-9da0-488d-b0a2-8706e14120a8 nodeName:}" failed. No retries permitted until 2026-05-28 15:12:24.47026303 +0000 UTC m=+96.769332519 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert") pod "ingress-canary-5lwgs" (UID: "37fbb52e-9da0-488d-b0a2-8706e14120a8") : secret "canary-serving-cert" not found
May 28 15:11:52.972769 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:52.972724    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:52.975938 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:52.975922    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 15:11:52.986210 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:52.986192    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 15:11:52.997615 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:52.997582    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqrq\" (UniqueName: \"kubernetes.io/projected/9884c711-a48a-419c-abc1-d50cf28d0b46-kube-api-access-hxqrq\") pod \"network-check-target-j5kk5\" (UID: \"9884c711-a48a-419c-abc1-d50cf28d0b46\") " pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:53.073602 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:53.073560    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:11:53.076078 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:53.076060    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 15:11:53.084099 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:53.084075    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 15:11:53.084175 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:11:53.084161    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs podName:60775ca8-b4ea-4176-bda3-58ad85bbc806 nodeName:}" failed. No retries permitted until 2026-05-28 15:12:57.084141789 +0000 UTC m=+129.383211281 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs") pod "network-metrics-daemon-9lnwm" (UID: "60775ca8-b4ea-4176-bda3-58ad85bbc806") : secret "metrics-daemon-secret" not found
May 28 15:11:53.106024 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:53.105999    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwwqm\""
May 28 15:11:53.114173 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:53.114152    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:53.303579 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:53.303541    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5kk5"]
May 28 15:11:53.304399 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:11:53.304375    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9884c711_a48a_419c_abc1_d50cf28d0b46.slice/crio-0d458ccb68a2617921e78fd9483fe57a5dba10817b3fba57daf0e33ea4e9cec0 WatchSource:0}: Error finding container 0d458ccb68a2617921e78fd9483fe57a5dba10817b3fba57daf0e33ea4e9cec0: Status 404 returned error can't find the container with id 0d458ccb68a2617921e78fd9483fe57a5dba10817b3fba57daf0e33ea4e9cec0
May 28 15:11:53.587593 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:53.587505    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5kk5" event={"ID":"9884c711-a48a-419c-abc1-d50cf28d0b46","Type":"ContainerStarted","Data":"0d458ccb68a2617921e78fd9483fe57a5dba10817b3fba57daf0e33ea4e9cec0"}
May 28 15:11:56.593913 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:56.593807    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5kk5" event={"ID":"9884c711-a48a-419c-abc1-d50cf28d0b46","Type":"ContainerStarted","Data":"da54a19fab462de46aa76aafed68aa85beffdab399d4d34e9b3f7e9015714f04"}
May 28 15:11:56.594348 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:56.593939    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:11:56.609778 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:11:56.609732    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j5kk5" podStartSLOduration=65.714860666 podStartE2EDuration="1m8.609716775s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:11:53.306113274 +0000 UTC m=+65.605182767" lastFinishedPulling="2026-05-28 15:11:56.200969383 +0000 UTC m=+68.500038876" observedRunningTime="2026-05-28 15:11:56.609278074 +0000 UTC m=+68.908347585" watchObservedRunningTime="2026-05-28 15:11:56.609716775 +0000 UTC m=+68.908786286"
May 28 15:12:24.378455 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:24.378398    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:12:24.378962 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:24.378538    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 15:12:24.378962 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:24.378598    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls podName:713bf691-c64b-485f-b8f8-7094daac71eb nodeName:}" failed. No retries permitted until 2026-05-28 15:13:28.378583316 +0000 UTC m=+160.677652805 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls") pod "dns-default-9n5pb" (UID: "713bf691-c64b-485f-b8f8-7094daac71eb") : secret "dns-default-metrics-tls" not found
May 28 15:12:24.479764 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:24.479727    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:12:24.479929 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:24.479876    2582 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 15:12:24.479973 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:24.479951    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert podName:37fbb52e-9da0-488d-b0a2-8706e14120a8 nodeName:}" failed. No retries permitted until 2026-05-28 15:13:28.479935534 +0000 UTC m=+160.779005023 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert") pod "ingress-canary-5lwgs" (UID: "37fbb52e-9da0-488d-b0a2-8706e14120a8") : secret "canary-serving-cert" not found
May 28 15:12:27.598067 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:27.598036    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j5kk5"
May 28 15:12:44.638831 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.638792    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-k6lc4"]
May 28 15:12:44.642083 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.642058    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5645c46bf9-mxj7s"]
May 28 15:12:44.642242 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.642224    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.645332 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.645314    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.648848 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.648825    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 28 15:12:44.650315 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.650292    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 28 15:12:44.650405 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.650297    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 28 15:12:44.650405 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.650368    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 28 15:12:44.650405 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.650298    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 28 15:12:44.650596 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.650569    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 28 15:12:44.650730 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.650693    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 28 15:12:44.651342 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.651329    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-glc25\""
May 28 15:12:44.651904 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.651890    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pfwm7\""
May 28 15:12:44.656185 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.656167    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5645c46bf9-mxj7s"]
May 28 15:12:44.664491 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.664461    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 28 15:12:44.667686 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.667660    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-k6lc4"]
May 28 15:12:44.669321 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.669301    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 28 15:12:44.719608 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719576    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/116f4a3b-549f-4ac0-97ad-b0269f62230d-ca-trust-extracted\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.719608 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-image-registry-private-configuration\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.719839 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-config\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.719839 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmr5b\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-kube-api-access-xmr5b\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.719839 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-bound-sa-token\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.719839 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-trusted-ca\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.719839 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-installation-pull-secrets\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.719839 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-serving-cert\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.720243 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.720243 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2f4s\" (UniqueName: \"kubernetes.io/projected/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-kube-api-access-c2f4s\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.720243 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-certificates\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.720243 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.719948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-trusted-ca\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.738402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.738370    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"]
May 28 15:12:44.740920 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.740906    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:44.743605 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.743584    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 28 15:12:44.743728 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.743588    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mg952\""
May 28 15:12:44.743837 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.743821    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 28 15:12:44.744947 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.744930    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 28 15:12:44.770143 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.762616    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"]
May 28 15:12:44.821249 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821216    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/116f4a3b-549f-4ac0-97ad-b0269f62230d-ca-trust-extracted\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.821443 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:44.821443 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821281    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-image-registry-private-configuration\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.821443 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821299    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-config\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.821443 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821347    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xmr5b\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-kube-api-access-xmr5b\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.821443 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821367    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-bound-sa-token\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.821443 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821383    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-trusted-ca\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.821443 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjqcz\" (UniqueName: \"kubernetes.io/projected/47d63f1b-668d-473b-a657-3bf3a3ac0a49-kube-api-access-qjqcz\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:44.821749 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821462    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-installation-pull-secrets\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.821749 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821617    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-serving-cert\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.821749 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821664    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.821749 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821692    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c2f4s\" (UniqueName: \"kubernetes.io/projected/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-kube-api-access-c2f4s\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.821749 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821744    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-certificates\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.821988 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821772    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-trusted-ca\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.821988 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:44.821853    2582 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 15:12:44.821988 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:44.821871    2582 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5645c46bf9-mxj7s: secret "image-registry-tls" not found
May 28 15:12:44.821988 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:44.821942    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls podName:116f4a3b-549f-4ac0-97ad-b0269f62230d nodeName:}" failed. No retries permitted until 2026-05-28 15:12:45.321921051 +0000 UTC m=+117.620990544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls") pod "image-registry-5645c46bf9-mxj7s" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d") : secret "image-registry-tls" not found
May 28 15:12:44.822206 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.821630    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/116f4a3b-549f-4ac0-97ad-b0269f62230d-ca-trust-extracted\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.822286 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.822258    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-config\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.822492 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.822471    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-trusted-ca\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.822684 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.822668    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-certificates\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.822721 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.822676    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-trusted-ca\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.823906 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.823886    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-serving-cert\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.824168 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.824148    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-installation-pull-secrets\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.824234 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.824157    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-image-registry-private-configuration\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.840674 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.840642    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-bound-sa-token\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.843698 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.843675    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2f4s\" (UniqueName: \"kubernetes.io/projected/d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d-kube-api-access-c2f4s\") pod \"console-operator-76dff48d59-k6lc4\" (UID: \"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d\") " pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:44.844031 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.844012    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmr5b\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-kube-api-access-xmr5b\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:44.922372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.922266    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:44.922530 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:44.922420    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 15:12:44.922530 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.922462    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qjqcz\" (UniqueName: \"kubernetes.io/projected/47d63f1b-668d-473b-a657-3bf3a3ac0a49-kube-api-access-qjqcz\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:44.922530 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:44.922494    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls podName:47d63f1b-668d-473b-a657-3bf3a3ac0a49 nodeName:}" failed. No retries permitted until 2026-05-28 15:12:45.422473594 +0000 UTC m=+117.721543088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls") pod "cluster-samples-operator-697587599-29nc7" (UID: "47d63f1b-668d-473b-a657-3bf3a3ac0a49") : secret "samples-operator-tls" not found
May 28 15:12:44.931017 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.930988    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjqcz\" (UniqueName: \"kubernetes.io/projected/47d63f1b-668d-473b-a657-3bf3a3ac0a49-kube-api-access-qjqcz\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:44.952917 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:44.952883    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:45.067013 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:45.066985    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-k6lc4"]
May 28 15:12:45.070076 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:12:45.070043    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bc276c_5d34_4b0e_83ea_a5ed8af32e8d.slice/crio-55a24af03b10a434f34537780a4c3feec7ff409935d913457357e6f6be1f2cda WatchSource:0}: Error finding container 55a24af03b10a434f34537780a4c3feec7ff409935d913457357e6f6be1f2cda: Status 404 returned error can't find the container with id 55a24af03b10a434f34537780a4c3feec7ff409935d913457357e6f6be1f2cda
May 28 15:12:45.326088 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:45.325992    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:45.326246 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:45.326171    2582 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 15:12:45.326246 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:45.326193    2582 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5645c46bf9-mxj7s: secret "image-registry-tls" not found
May 28 15:12:45.326246 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:45.326246    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls podName:116f4a3b-549f-4ac0-97ad-b0269f62230d nodeName:}" failed. No retries permitted until 2026-05-28 15:12:46.32623189 +0000 UTC m=+118.625301379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls") pod "image-registry-5645c46bf9-mxj7s" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d") : secret "image-registry-tls" not found
May 28 15:12:45.427020 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:45.426987    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:45.427200 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:45.427176    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 15:12:45.427273 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:45.427262    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls podName:47d63f1b-668d-473b-a657-3bf3a3ac0a49 nodeName:}" failed. No retries permitted until 2026-05-28 15:12:46.427239265 +0000 UTC m=+118.726308759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls") pod "cluster-samples-operator-697587599-29nc7" (UID: "47d63f1b-668d-473b-a657-3bf3a3ac0a49") : secret "samples-operator-tls" not found
May 28 15:12:45.680549 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:45.680508    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" event={"ID":"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d","Type":"ContainerStarted","Data":"55a24af03b10a434f34537780a4c3feec7ff409935d913457357e6f6be1f2cda"}
May 28 15:12:46.335614 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:46.335578    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:46.335779 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:46.335720    2582 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 15:12:46.335779 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:46.335737    2582 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5645c46bf9-mxj7s: secret "image-registry-tls" not found
May 28 15:12:46.335857 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:46.335795    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls podName:116f4a3b-549f-4ac0-97ad-b0269f62230d nodeName:}" failed. No retries permitted until 2026-05-28 15:12:48.335779792 +0000 UTC m=+120.634849286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls") pod "image-registry-5645c46bf9-mxj7s" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d") : secret "image-registry-tls" not found
May 28 15:12:46.436551 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:46.436515    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:46.436733 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:46.436688    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 15:12:46.436786 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:46.436766    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls podName:47d63f1b-668d-473b-a657-3bf3a3ac0a49 nodeName:}" failed. No retries permitted until 2026-05-28 15:12:48.436746599 +0000 UTC m=+120.735816099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls") pod "cluster-samples-operator-697587599-29nc7" (UID: "47d63f1b-668d-473b-a657-3bf3a3ac0a49") : secret "samples-operator-tls" not found
May 28 15:12:47.685848 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:47.685821    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/0.log"
May 28 15:12:47.686298 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:47.685858    2582 generic.go:358] "Generic (PLEG): container finished" podID="d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d" containerID="c5acbc13c2417a0a21805927ae8ad604911299a861f9dbcaf6701d1bfa304151" exitCode=255
May 28 15:12:47.686298 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:47.685885    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" event={"ID":"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d","Type":"ContainerDied","Data":"c5acbc13c2417a0a21805927ae8ad604911299a861f9dbcaf6701d1bfa304151"}
May 28 15:12:47.686298 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:47.686135    2582 scope.go:117] "RemoveContainer" containerID="c5acbc13c2417a0a21805927ae8ad604911299a861f9dbcaf6701d1bfa304151"
May 28 15:12:48.353319 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:48.353256    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:48.353493 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:48.353394    2582 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 15:12:48.353493 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:48.353413    2582 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5645c46bf9-mxj7s: secret "image-registry-tls" not found
May 28 15:12:48.353493 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:48.353478    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls podName:116f4a3b-549f-4ac0-97ad-b0269f62230d nodeName:}" failed. No retries permitted until 2026-05-28 15:12:52.353461835 +0000 UTC m=+124.652531324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls") pod "image-registry-5645c46bf9-mxj7s" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d") : secret "image-registry-tls" not found
May 28 15:12:48.453565 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:48.453538    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:48.453690 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:48.453674    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 15:12:48.453753 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:48.453745    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls podName:47d63f1b-668d-473b-a657-3bf3a3ac0a49 nodeName:}" failed. No retries permitted until 2026-05-28 15:12:52.453728893 +0000 UTC m=+124.752798382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls") pod "cluster-samples-operator-697587599-29nc7" (UID: "47d63f1b-668d-473b-a657-3bf3a3ac0a49") : secret "samples-operator-tls" not found
May 28 15:12:48.689040 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:48.689019    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/1.log"
May 28 15:12:48.689409 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:48.689392    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/0.log"
May 28 15:12:48.689458 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:48.689429    2582 generic.go:358] "Generic (PLEG): container finished" podID="d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d" containerID="9169e5a50492dc445687068ec67f090291ba224c7f918b905faeed59d9604efc" exitCode=255
May 28 15:12:48.689494 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:48.689470    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" event={"ID":"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d","Type":"ContainerDied","Data":"9169e5a50492dc445687068ec67f090291ba224c7f918b905faeed59d9604efc"}
May 28 15:12:48.689526 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:48.689496    2582 scope.go:117] "RemoveContainer" containerID="c5acbc13c2417a0a21805927ae8ad604911299a861f9dbcaf6701d1bfa304151"
May 28 15:12:48.689759 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:48.689734    2582 scope.go:117] "RemoveContainer" containerID="9169e5a50492dc445687068ec67f090291ba224c7f918b905faeed59d9604efc"
May 28 15:12:48.689930 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:48.689910    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-k6lc4_openshift-console-operator(d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d)\"" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" podUID="d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d"
May 28 15:12:49.692318 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.692289    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/1.log"
May 28 15:12:49.692778 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.692691    2582 scope.go:117] "RemoveContainer" containerID="9169e5a50492dc445687068ec67f090291ba224c7f918b905faeed59d9604efc"
May 28 15:12:49.692914 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:49.692894    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-k6lc4_openshift-console-operator(d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d)\"" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" podUID="d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d"
May 28 15:12:49.714780 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.714752    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6"]
May 28 15:12:49.717913 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.717899    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6"
May 28 15:12:49.720135 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.720095    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 28 15:12:49.720249 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.720187    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 28 15:12:49.720249 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.720189    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cqn6v\""
May 28 15:12:49.726044 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.726024    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6"]
May 28 15:12:49.863919 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.863882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5f9\" (UniqueName: \"kubernetes.io/projected/979677a1-5c8c-4286-aa13-3d92aab72251-kube-api-access-vk5f9\") pod \"migrator-b985f5d89-85lk6\" (UID: \"979677a1-5c8c-4286-aa13-3d92aab72251\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6"
May 28 15:12:49.965266 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.965198    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5f9\" (UniqueName: \"kubernetes.io/projected/979677a1-5c8c-4286-aa13-3d92aab72251-kube-api-access-vk5f9\") pod \"migrator-b985f5d89-85lk6\" (UID: \"979677a1-5c8c-4286-aa13-3d92aab72251\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6"
May 28 15:12:49.973544 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:49.973519    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5f9\" (UniqueName: \"kubernetes.io/projected/979677a1-5c8c-4286-aa13-3d92aab72251-kube-api-access-vk5f9\") pod \"migrator-b985f5d89-85lk6\" (UID: \"979677a1-5c8c-4286-aa13-3d92aab72251\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6"
May 28 15:12:50.026815 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:50.026785    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6"
May 28 15:12:50.144906 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:50.144874    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6"]
May 28 15:12:50.147687 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:12:50.147663    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979677a1_5c8c_4286_aa13_3d92aab72251.slice/crio-c9de52aa154c9688ad92135767f21d5eedfd7344e149b21267ea77d0b054580b WatchSource:0}: Error finding container c9de52aa154c9688ad92135767f21d5eedfd7344e149b21267ea77d0b054580b: Status 404 returned error can't find the container with id c9de52aa154c9688ad92135767f21d5eedfd7344e149b21267ea77d0b054580b
May 28 15:12:50.695777 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:50.695731    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6" event={"ID":"979677a1-5c8c-4286-aa13-3d92aab72251","Type":"ContainerStarted","Data":"c9de52aa154c9688ad92135767f21d5eedfd7344e149b21267ea77d0b054580b"}
May 28 15:12:51.617866 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.617842    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sd8jw_f11838e8-88b0-4035-85e4-99fd5e9026c3/dns-node-resolver/0.log"
May 28 15:12:51.699324 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.699294    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6" event={"ID":"979677a1-5c8c-4286-aa13-3d92aab72251","Type":"ContainerStarted","Data":"43b0c6aa9b7a67e1e37fe4a45d8a222a004626b4349999b071910151c9b0e3d0"}
May 28 15:12:51.699654 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.699332    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6" event={"ID":"979677a1-5c8c-4286-aa13-3d92aab72251","Type":"ContainerStarted","Data":"32247bf1e3ee236c2c13452332fb143dfefba35cc103f0c5fdc9bc0f1541af27"}
May 28 15:12:51.717027 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.716979    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-85lk6" podStartSLOduration=1.596362988 podStartE2EDuration="2.716961254s" podCreationTimestamp="2026-05-28 15:12:49 +0000 UTC" firstStartedPulling="2026-05-28 15:12:50.149358216 +0000 UTC m=+122.448427705" lastFinishedPulling="2026-05-28 15:12:51.269956478 +0000 UTC m=+123.569025971" observedRunningTime="2026-05-28 15:12:51.716575648 +0000 UTC m=+124.015645160" watchObservedRunningTime="2026-05-28 15:12:51.716961254 +0000 UTC m=+124.016030766"
May 28 15:12:51.744336 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.744304    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-kqzcs"]
May 28 15:12:51.746897 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.746883    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:51.749338 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.749317    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 28 15:12:51.749444 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.749359    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 28 15:12:51.749589 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.749574    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 28 15:12:51.749677 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.749659    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cfnwd\""
May 28 15:12:51.749722 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.749665    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 28 15:12:51.755264 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.755243    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-kqzcs"]
May 28 15:12:51.880460 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.880372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eea18e20-42f1-488e-a0a5-72af0baebc09-signing-key\") pod \"service-ca-67d667c7d-kqzcs\" (UID: \"eea18e20-42f1-488e-a0a5-72af0baebc09\") " pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:51.880460 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.880430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eea18e20-42f1-488e-a0a5-72af0baebc09-signing-cabundle\") pod \"service-ca-67d667c7d-kqzcs\" (UID: \"eea18e20-42f1-488e-a0a5-72af0baebc09\") " pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:51.880639 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.880480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c86s\" (UniqueName: \"kubernetes.io/projected/eea18e20-42f1-488e-a0a5-72af0baebc09-kube-api-access-2c86s\") pod \"service-ca-67d667c7d-kqzcs\" (UID: \"eea18e20-42f1-488e-a0a5-72af0baebc09\") " pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:51.981425 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.981397    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2c86s\" (UniqueName: \"kubernetes.io/projected/eea18e20-42f1-488e-a0a5-72af0baebc09-kube-api-access-2c86s\") pod \"service-ca-67d667c7d-kqzcs\" (UID: \"eea18e20-42f1-488e-a0a5-72af0baebc09\") " pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:51.981523 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.981450    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eea18e20-42f1-488e-a0a5-72af0baebc09-signing-key\") pod \"service-ca-67d667c7d-kqzcs\" (UID: \"eea18e20-42f1-488e-a0a5-72af0baebc09\") " pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:51.981523 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.981490    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eea18e20-42f1-488e-a0a5-72af0baebc09-signing-cabundle\") pod \"service-ca-67d667c7d-kqzcs\" (UID: \"eea18e20-42f1-488e-a0a5-72af0baebc09\") " pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:51.982038 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.982023    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eea18e20-42f1-488e-a0a5-72af0baebc09-signing-cabundle\") pod \"service-ca-67d667c7d-kqzcs\" (UID: \"eea18e20-42f1-488e-a0a5-72af0baebc09\") " pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:51.983774 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.983756    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eea18e20-42f1-488e-a0a5-72af0baebc09-signing-key\") pod \"service-ca-67d667c7d-kqzcs\" (UID: \"eea18e20-42f1-488e-a0a5-72af0baebc09\") " pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:51.990782 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:51.990758    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c86s\" (UniqueName: \"kubernetes.io/projected/eea18e20-42f1-488e-a0a5-72af0baebc09-kube-api-access-2c86s\") pod \"service-ca-67d667c7d-kqzcs\" (UID: \"eea18e20-42f1-488e-a0a5-72af0baebc09\") " pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:52.055782 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:52.055746    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-kqzcs"
May 28 15:12:52.171489 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:52.171417    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-kqzcs"]
May 28 15:12:52.174430 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:12:52.174405    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea18e20_42f1_488e_a0a5_72af0baebc09.slice/crio-c0097cba36f80ba8bd7c9e8ffa8803ae98874400c68e9788a7f4268d3d23c541 WatchSource:0}: Error finding container c0097cba36f80ba8bd7c9e8ffa8803ae98874400c68e9788a7f4268d3d23c541: Status 404 returned error can't find the container with id c0097cba36f80ba8bd7c9e8ffa8803ae98874400c68e9788a7f4268d3d23c541
May 28 15:12:52.207237 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:52.207214    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjtv8_2fe05b8d-d2bb-40c3-9642-61b447da82ab/node-ca/0.log"
May 28 15:12:52.384671 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:52.384635    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:12:52.384847 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:52.384771    2582 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 15:12:52.384847 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:52.384787    2582 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5645c46bf9-mxj7s: secret "image-registry-tls" not found
May 28 15:12:52.384847 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:52.384836    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls podName:116f4a3b-549f-4ac0-97ad-b0269f62230d nodeName:}" failed. No retries permitted until 2026-05-28 15:13:00.384821972 +0000 UTC m=+132.683891466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls") pod "image-registry-5645c46bf9-mxj7s" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d") : secret "image-registry-tls" not found
May 28 15:12:52.485363 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:52.485272    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:12:52.485487 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:52.485417    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 15:12:52.485531 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:52.485491    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls podName:47d63f1b-668d-473b-a657-3bf3a3ac0a49 nodeName:}" failed. No retries permitted until 2026-05-28 15:13:00.485474997 +0000 UTC m=+132.784544486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls") pod "cluster-samples-operator-697587599-29nc7" (UID: "47d63f1b-668d-473b-a657-3bf3a3ac0a49") : secret "samples-operator-tls" not found
May 28 15:12:52.702302 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:52.702268    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-kqzcs" event={"ID":"eea18e20-42f1-488e-a0a5-72af0baebc09","Type":"ContainerStarted","Data":"c0097cba36f80ba8bd7c9e8ffa8803ae98874400c68e9788a7f4268d3d23c541"}
May 28 15:12:54.707718 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:54.707679    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-kqzcs" event={"ID":"eea18e20-42f1-488e-a0a5-72af0baebc09","Type":"ContainerStarted","Data":"960b72aff6d9868d9a4c05ed21692a35bc49a8ca6602a0c2bc9e2de277ff3c90"}
May 28 15:12:54.727867 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:54.727819    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-kqzcs" podStartSLOduration=2.003414762 podStartE2EDuration="3.727806684s" podCreationTimestamp="2026-05-28 15:12:51 +0000 UTC" firstStartedPulling="2026-05-28 15:12:52.176079953 +0000 UTC m=+124.475149442" lastFinishedPulling="2026-05-28 15:12:53.900471868 +0000 UTC m=+126.199541364" observedRunningTime="2026-05-28 15:12:54.726202818 +0000 UTC m=+127.025272328" watchObservedRunningTime="2026-05-28 15:12:54.727806684 +0000 UTC m=+127.026876194"
May 28 15:12:54.953261 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:54.953225    2582 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:54.953261 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:54.953261    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:12:54.953645 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:54.953633    2582 scope.go:117] "RemoveContainer" containerID="9169e5a50492dc445687068ec67f090291ba224c7f918b905faeed59d9604efc"
May 28 15:12:54.953814 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:54.953798    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-k6lc4_openshift-console-operator(d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d)\"" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" podUID="d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d"
May 28 15:12:57.125779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:12:57.125740    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:12:57.126285 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:57.125859    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 15:12:57.126285 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:12:57.125918    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs podName:60775ca8-b4ea-4176-bda3-58ad85bbc806 nodeName:}" failed. No retries permitted until 2026-05-28 15:14:59.125904609 +0000 UTC m=+251.424974098 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs") pod "network-metrics-daemon-9lnwm" (UID: "60775ca8-b4ea-4176-bda3-58ad85bbc806") : secret "metrics-daemon-secret" not found
May 28 15:13:00.446154 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.446108    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:13:00.448424 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.448404    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") pod \"image-registry-5645c46bf9-mxj7s\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") " pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:13:00.546784 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.546758    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:13:00.548838 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.548812    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/47d63f1b-668d-473b-a657-3bf3a3ac0a49-samples-operator-tls\") pod \"cluster-samples-operator-697587599-29nc7\" (UID: \"47d63f1b-668d-473b-a657-3bf3a3ac0a49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:13:00.559312 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.559295    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-glc25\""
May 28 15:13:00.567462 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.567448    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:13:00.654473 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.654448    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mg952\""
May 28 15:13:00.660790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.660762    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"
May 28 15:13:00.688736 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.688697    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5645c46bf9-mxj7s"]
May 28 15:13:00.692270 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:00.692239    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116f4a3b_549f_4ac0_97ad_b0269f62230d.slice/crio-3e6a1f53cd6234dd718ad49c1b9dd0456015e8863a5ff2974b69d4d0a58b9e15 WatchSource:0}: Error finding container 3e6a1f53cd6234dd718ad49c1b9dd0456015e8863a5ff2974b69d4d0a58b9e15: Status 404 returned error can't find the container with id 3e6a1f53cd6234dd718ad49c1b9dd0456015e8863a5ff2974b69d4d0a58b9e15
May 28 15:13:00.719362 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.719333    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s" event={"ID":"116f4a3b-549f-4ac0-97ad-b0269f62230d","Type":"ContainerStarted","Data":"3e6a1f53cd6234dd718ad49c1b9dd0456015e8863a5ff2974b69d4d0a58b9e15"}
May 28 15:13:00.784647 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:00.784623    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7"]
May 28 15:13:01.724791 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:01.724750    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s" event={"ID":"116f4a3b-549f-4ac0-97ad-b0269f62230d","Type":"ContainerStarted","Data":"12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b"}
May 28 15:13:01.725248 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:01.724904    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:13:01.725795 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:01.725768    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7" event={"ID":"47d63f1b-668d-473b-a657-3bf3a3ac0a49","Type":"ContainerStarted","Data":"c09e60fc9f5cbf53b1b8cc4c0f3008cd0ff0c1291e5f0b95323f633816d5a64f"}
May 28 15:13:01.747040 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:01.747002    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s" podStartSLOduration=17.746989459 podStartE2EDuration="17.746989459s" podCreationTimestamp="2026-05-28 15:12:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:13:01.745735313 +0000 UTC m=+134.044804837" watchObservedRunningTime="2026-05-28 15:13:01.746989459 +0000 UTC m=+134.046058970"
May 28 15:13:02.730170 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:02.730140    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7" event={"ID":"47d63f1b-668d-473b-a657-3bf3a3ac0a49","Type":"ContainerStarted","Data":"41463a6588970aea4c8f66c135cbce4934255a3010b9b9aca9624ae49e5e7943"}
May 28 15:13:02.730511 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:02.730181    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7" event={"ID":"47d63f1b-668d-473b-a657-3bf3a3ac0a49","Type":"ContainerStarted","Data":"3d62fe9668e25e795259ffa4db0a587b1620e536e658f620c8449fd8e8b4ad5c"}
May 28 15:13:02.748143 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:02.747952    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-29nc7" podStartSLOduration=16.952585827 podStartE2EDuration="18.747937498s" podCreationTimestamp="2026-05-28 15:12:44 +0000 UTC" firstStartedPulling="2026-05-28 15:13:00.817036612 +0000 UTC m=+133.116106104" lastFinishedPulling="2026-05-28 15:13:02.612388269 +0000 UTC m=+134.911457775" observedRunningTime="2026-05-28 15:13:02.746983852 +0000 UTC m=+135.046053364" watchObservedRunningTime="2026-05-28 15:13:02.747937498 +0000 UTC m=+135.047007009"
May 28 15:13:09.288293 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:09.288262    2582 scope.go:117] "RemoveContainer" containerID="9169e5a50492dc445687068ec67f090291ba224c7f918b905faeed59d9604efc"
May 28 15:13:09.747134 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:09.747102    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:13:09.747444 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:09.747427    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/1.log"
May 28 15:13:09.747551 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:09.747464    2582 generic.go:358] "Generic (PLEG): container finished" podID="d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d" containerID="a00f5787af3b50ad9ef81446de3c14182bbb984ea1d1aea78da41205efe0470f" exitCode=255
May 28 15:13:09.747551 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:09.747539    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" event={"ID":"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d","Type":"ContainerDied","Data":"a00f5787af3b50ad9ef81446de3c14182bbb984ea1d1aea78da41205efe0470f"}
May 28 15:13:09.747625 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:09.747577    2582 scope.go:117] "RemoveContainer" containerID="9169e5a50492dc445687068ec67f090291ba224c7f918b905faeed59d9604efc"
May 28 15:13:09.747870 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:09.747843    2582 scope.go:117] "RemoveContainer" containerID="a00f5787af3b50ad9ef81446de3c14182bbb984ea1d1aea78da41205efe0470f"
May 28 15:13:09.748035 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:09.748007    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-k6lc4_openshift-console-operator(d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d)\"" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" podUID="d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d"
May 28 15:13:10.751103 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:10.751077    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:13:14.953371 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:14.953345    2582 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:13:14.953371 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:14.953375    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:13:14.953751 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:14.953642    2582 scope.go:117] "RemoveContainer" containerID="a00f5787af3b50ad9ef81446de3c14182bbb984ea1d1aea78da41205efe0470f"
May 28 15:13:14.953807 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:14.953790    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-k6lc4_openshift-console-operator(d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d)\"" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" podUID="d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d"
May 28 15:13:15.295802 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.295728    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5645c46bf9-mxj7s"]
May 28 15:13:15.299614 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.299587    2582 patch_prober.go:28] interesting pod/image-registry-5645c46bf9-mxj7s container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 28 15:13:15.299736 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.299628    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s" podUID="116f4a3b-549f-4ac0-97ad-b0269f62230d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 28 15:13:15.451607 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.451582    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vg8zf"]
May 28 15:13:15.455658 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.455641    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.459074 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.459054    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 28 15:13:15.459167 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.459092    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 28 15:13:15.459209 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.459182    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tqbxg\""
May 28 15:13:15.462024 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.462007    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 28 15:13:15.462099 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.462062    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 28 15:13:15.474380 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.474361    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vg8zf"]
May 28 15:13:15.556172 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.556099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-data-volume\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.556172 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.556171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fw6\" (UniqueName: \"kubernetes.io/projected/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-kube-api-access-62fw6\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.556320 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.556281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.556320 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.556308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-crio-socket\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.556408 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.556328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.657079 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.657054    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.657220 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.657088    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-crio-socket\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.657220 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.657116    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.657220 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.657154    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-data-volume\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.657220 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.657196    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-62fw6\" (UniqueName: \"kubernetes.io/projected/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-kube-api-access-62fw6\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.657220 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.657199    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-crio-socket\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.657545 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.657522    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-data-volume\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.657653 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.657573    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.659202 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.659183    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.670218 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.670199    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fw6\" (UniqueName: \"kubernetes.io/projected/cfab63fc-ed9e-4984-b0f7-e0afd45b74bf-kube-api-access-62fw6\") pod \"insights-runtime-extractor-vg8zf\" (UID: \"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf\") " pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.763869 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.763844    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vg8zf"
May 28 15:13:15.883998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:15.883970    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vg8zf"]
May 28 15:13:15.887373 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:15.887338    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfab63fc_ed9e_4984_b0f7_e0afd45b74bf.slice/crio-fda894d7b4ab00af0a415d28115aed7824d879d2f39f6ee9eb79bf37eb3b786c WatchSource:0}: Error finding container fda894d7b4ab00af0a415d28115aed7824d879d2f39f6ee9eb79bf37eb3b786c: Status 404 returned error can't find the container with id fda894d7b4ab00af0a415d28115aed7824d879d2f39f6ee9eb79bf37eb3b786c
May 28 15:13:16.769798 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:16.769772    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vg8zf" event={"ID":"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf","Type":"ContainerStarted","Data":"a8312282dfe545b3bed65dae4f94766e0ce7fe7353ff296d98253ef4bfb7b804"}
May 28 15:13:16.770086 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:16.769807    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vg8zf" event={"ID":"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf","Type":"ContainerStarted","Data":"36a840479d01efb2410b3167d96c8d72553d1894a9a66bfb2c5ae2deb5380d1c"}
May 28 15:13:16.770086 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:16.769817    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vg8zf" event={"ID":"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf","Type":"ContainerStarted","Data":"fda894d7b4ab00af0a415d28115aed7824d879d2f39f6ee9eb79bf37eb3b786c"}
May 28 15:13:18.776045 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:18.776014    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vg8zf" event={"ID":"cfab63fc-ed9e-4984-b0f7-e0afd45b74bf","Type":"ContainerStarted","Data":"d2d5d6ce22a742cb0b6fe00cc7ce587b891193923f8bd46b734067fbec26294b"}
May 28 15:13:18.794019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:18.793974    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vg8zf" podStartSLOduration=1.847905494 podStartE2EDuration="3.793960729s" podCreationTimestamp="2026-05-28 15:13:15 +0000 UTC" firstStartedPulling="2026-05-28 15:13:15.942540975 +0000 UTC m=+148.241610464" lastFinishedPulling="2026-05-28 15:13:17.888596209 +0000 UTC m=+150.187665699" observedRunningTime="2026-05-28 15:13:18.793908263 +0000 UTC m=+151.092977774" watchObservedRunningTime="2026-05-28 15:13:18.793960729 +0000 UTC m=+151.093030241"
May 28 15:13:19.367636 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.367595    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"]
May 28 15:13:19.370596 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.370580    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"
May 28 15:13:19.372881 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.372857    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 28 15:13:19.372988 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.372937    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7dhpv\""
May 28 15:13:19.381005 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.380985    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"]
May 28 15:13:19.486043 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.486000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/41ec3631-f453-4883-8bb9-5954671f1a73-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-2kl74\" (UID: \"41ec3631-f453-4883-8bb9-5954671f1a73\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"
May 28 15:13:19.587060 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.587029    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/41ec3631-f453-4883-8bb9-5954671f1a73-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-2kl74\" (UID: \"41ec3631-f453-4883-8bb9-5954671f1a73\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"
May 28 15:13:19.589346 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.589328    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/41ec3631-f453-4883-8bb9-5954671f1a73-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-2kl74\" (UID: \"41ec3631-f453-4883-8bb9-5954671f1a73\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"
May 28 15:13:19.680987 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.680897    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"
May 28 15:13:19.795034 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:19.795007    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"]
May 28 15:13:19.797954 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:19.797927    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ec3631_f453_4883_8bb9_5954671f1a73.slice/crio-debd6b7585f7c38a143d73bfa038bf3480feb593e3857875e0016be010fc543f WatchSource:0}: Error finding container debd6b7585f7c38a143d73bfa038bf3480feb593e3857875e0016be010fc543f: Status 404 returned error can't find the container with id debd6b7585f7c38a143d73bfa038bf3480feb593e3857875e0016be010fc543f
May 28 15:13:20.784869 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:20.784838    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74" event={"ID":"41ec3631-f453-4883-8bb9-5954671f1a73","Type":"ContainerStarted","Data":"debd6b7585f7c38a143d73bfa038bf3480feb593e3857875e0016be010fc543f"}
May 28 15:13:21.788184 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:21.788144    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74" event={"ID":"41ec3631-f453-4883-8bb9-5954671f1a73","Type":"ContainerStarted","Data":"17e708323fdf26403c36563c9d4c30b068958a2770f4845b47a2e3cb99956a5e"}
May 28 15:13:21.788633 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:21.788341    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"
May 28 15:13:21.792700 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:21.792681    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74"
May 28 15:13:21.803505 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:21.803466    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-2kl74" podStartSLOduration=1.7992390280000001 podStartE2EDuration="2.803453912s" podCreationTimestamp="2026-05-28 15:13:19 +0000 UTC" firstStartedPulling="2026-05-28 15:13:19.799717914 +0000 UTC m=+152.098787410" lastFinishedPulling="2026-05-28 15:13:20.803932791 +0000 UTC m=+153.103002294" observedRunningTime="2026-05-28 15:13:21.803280732 +0000 UTC m=+154.102350271" watchObservedRunningTime="2026-05-28 15:13:21.803453912 +0000 UTC m=+154.102523425"
May 28 15:13:22.445512 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.445463    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"]
May 28 15:13:22.481422 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.481390    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"]
May 28 15:13:22.481565 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.481542    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.484416 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.484394    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 28 15:13:22.484416 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.484404    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 28 15:13:22.484603 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.484350    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 28 15:13:22.484603 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.484459    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 28 15:13:22.485653 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.485627    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zktq4\""
May 28 15:13:22.485768 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.485699    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 28 15:13:22.605505 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.605457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td89p\" (UniqueName: \"kubernetes.io/projected/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-kube-api-access-td89p\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.605690 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.605514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.605690 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.605540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.605690 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.605564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.705992 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.705906    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-td89p\" (UniqueName: \"kubernetes.io/projected/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-kube-api-access-td89p\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.705992 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.705954    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.705992 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.705983    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.706225 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.706014    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.706730 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.706704    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.708364 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.708342    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.708451 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.708428    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.716234 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.716214    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td89p\" (UniqueName: \"kubernetes.io/projected/b80c2e55-3fb2-4a4e-bcd7-d59323fd6224-kube-api-access-td89p\") pod \"prometheus-operator-7779bbdd78-4tbcl\" (UID: \"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.789688 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.789661    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"
May 28 15:13:22.910664 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:22.910634    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl"]
May 28 15:13:22.913375 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:22.913346    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80c2e55_3fb2_4a4e_bcd7_d59323fd6224.slice/crio-cbd00547b3f200cb4d2d1cd6f2c736d77ac61fc868733df0a6d850a4dd3a46ca WatchSource:0}: Error finding container cbd00547b3f200cb4d2d1cd6f2c736d77ac61fc868733df0a6d850a4dd3a46ca: Status 404 returned error can't find the container with id cbd00547b3f200cb4d2d1cd6f2c736d77ac61fc868733df0a6d850a4dd3a46ca
May 28 15:13:23.591860 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:23.591821    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9n5pb" podUID="713bf691-c64b-485f-b8f8-7094daac71eb"
May 28 15:13:23.601052 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:23.601008    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-5lwgs" podUID="37fbb52e-9da0-488d-b0a2-8706e14120a8"
May 28 15:13:23.794764 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:23.794720    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl" event={"ID":"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224","Type":"ContainerStarted","Data":"cbd00547b3f200cb4d2d1cd6f2c736d77ac61fc868733df0a6d850a4dd3a46ca"}
May 28 15:13:23.794764 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:23.794751    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9n5pb"
May 28 15:13:24.299612 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:24.299583    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9lnwm" podUID="60775ca8-b4ea-4176-bda3-58ad85bbc806"
May 28 15:13:24.800432 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:24.800397    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl" event={"ID":"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224","Type":"ContainerStarted","Data":"a4d2b045fa2923c02c3f68f138d2c97aefd66fcb063e1c4e9668573264ab3ce3"}
May 28 15:13:24.800432 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:24.800433    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl" event={"ID":"b80c2e55-3fb2-4a4e-bcd7-d59323fd6224","Type":"ContainerStarted","Data":"17c414a26fa529ce22003c5f5f99ac2593bdc8e24a7567e4389c67c284972a25"}
May 28 15:13:24.822444 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:24.822395    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4tbcl" podStartSLOduration=1.557570889 podStartE2EDuration="2.822380441s" podCreationTimestamp="2026-05-28 15:13:22 +0000 UTC" firstStartedPulling="2026-05-28 15:13:22.915063054 +0000 UTC m=+155.214132543" lastFinishedPulling="2026-05-28 15:13:24.179872603 +0000 UTC m=+156.478942095" observedRunningTime="2026-05-28 15:13:24.821932682 +0000 UTC m=+157.121002192" watchObservedRunningTime="2026-05-28 15:13:24.822380441 +0000 UTC m=+157.121449952"
May 28 15:13:25.299592 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:25.299561    2582 patch_prober.go:28] interesting pod/image-registry-5645c46bf9-mxj7s container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 28 15:13:25.299734 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:25.299609    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s" podUID="116f4a3b-549f-4ac0-97ad-b0269f62230d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 28 15:13:26.288101 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.288027    2582 scope.go:117] "RemoveContainer" containerID="a00f5787af3b50ad9ef81446de3c14182bbb984ea1d1aea78da41205efe0470f"
May 28 15:13:26.288472 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:26.288199    2582 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-k6lc4_openshift-console-operator(d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d)\"" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" podUID="d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d"
May 28 15:13:26.895100 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.895063    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kkd7p"]
May 28 15:13:26.898424 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.898407    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:26.921061 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.921033    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 28 15:13:26.921249 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.921227    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 28 15:13:26.921321 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.921302    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 28 15:13:26.921541 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.921528    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ggxkp\""
May 28 15:13:26.940905 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.940869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a131cb29-6526-475f-b894-1bcedb93b59a-metrics-client-ca\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:26.941013 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.940920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-wtmp\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:26.941013 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.940942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwld5\" (UniqueName: \"kubernetes.io/projected/a131cb29-6526-475f-b894-1bcedb93b59a-kube-api-access-dwld5\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:26.941013 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.940998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a131cb29-6526-475f-b894-1bcedb93b59a-sys\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:26.941137 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.941043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-textfile\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:26.941137 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.941068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-tls\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:26.941137 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.941104    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:26.941242 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.941143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a131cb29-6526-475f-b894-1bcedb93b59a-root\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:26.941242 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:26.941168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.041802 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.041773    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.041967 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.041847    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a131cb29-6526-475f-b894-1bcedb93b59a-metrics-client-ca\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.041967 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.041880    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-wtmp\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.041967 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.041905    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dwld5\" (UniqueName: \"kubernetes.io/projected/a131cb29-6526-475f-b894-1bcedb93b59a-kube-api-access-dwld5\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.041967 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.041939    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a131cb29-6526-475f-b894-1bcedb93b59a-sys\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042198 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.041985    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-textfile\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042198 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.042009    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-tls\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042198 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.042055    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042198 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.042067    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-wtmp\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042198 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.042079    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a131cb29-6526-475f-b894-1bcedb93b59a-root\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042198 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.042078    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a131cb29-6526-475f-b894-1bcedb93b59a-sys\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042198 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.042157    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a131cb29-6526-475f-b894-1bcedb93b59a-root\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042198 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:27.042198    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 28 15:13:27.042566 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:27.042291    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-tls podName:a131cb29-6526-475f-b894-1bcedb93b59a nodeName:}" failed. No retries permitted until 2026-05-28 15:13:27.542269076 +0000 UTC m=+159.841338578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-tls") pod "node-exporter-kkd7p" (UID: "a131cb29-6526-475f-b894-1bcedb93b59a") : secret "node-exporter-tls" not found
May 28 15:13:27.042626 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.042564    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-textfile\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042779 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.042761    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a131cb29-6526-475f-b894-1bcedb93b59a-metrics-client-ca\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.042867 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.042845    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.044237 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.044220    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.058354 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.058332    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwld5\" (UniqueName: \"kubernetes.io/projected/a131cb29-6526-475f-b894-1bcedb93b59a-kube-api-access-dwld5\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.545457 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.545418    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-tls\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.547685 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.547668    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a131cb29-6526-475f-b894-1bcedb93b59a-node-exporter-tls\") pod \"node-exporter-kkd7p\" (UID: \"a131cb29-6526-475f-b894-1bcedb93b59a\") " pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.809250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.809169    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kkd7p"
May 28 15:13:27.817905 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:27.817875    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda131cb29_6526_475f_b894_1bcedb93b59a.slice/crio-d4d9818d2980b2f0ca3748f5fc00201c458532ab17f827dc6824e90c2dc9c9ff WatchSource:0}: Error finding container d4d9818d2980b2f0ca3748f5fc00201c458532ab17f827dc6824e90c2dc9c9ff: Status 404 returned error can't find the container with id d4d9818d2980b2f0ca3748f5fc00201c458532ab17f827dc6824e90c2dc9c9ff
May 28 15:13:27.962418 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.962382    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 15:13:27.967110 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.967089    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:27.969606 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.969577    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 28 15:13:27.970062 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.969849    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 28 15:13:27.970566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.970249    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 28 15:13:27.970566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.970332    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 28 15:13:27.970566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.970355    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 28 15:13:27.970566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.970445    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6vtss\""
May 28 15:13:27.970566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.970453    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 28 15:13:27.970566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.970490    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 28 15:13:27.970566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.970534    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 28 15:13:27.970867 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.970574    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 28 15:13:27.978407 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:27.978370    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 15:13:28.050473 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-web-config\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050473 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050660 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050660 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050660 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050660 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050660 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050660 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r75rm\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-kube-api-access-r75rm\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050660 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050660 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-out\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050891 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050891 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.050891 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.050777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.151946 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.151913    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.151946 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.151949    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-web-config\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152225 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.151966    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152225 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152161    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152225 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152193    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152258    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152294    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152324    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152347    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r75rm\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-kube-api-access-r75rm\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152576 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152382    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152576 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152418    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-out\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152576 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152447    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.152576 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.152505    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.153785 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:28.153619    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-trusted-ca-bundle podName:c9422d2e-f331-4f5d-943a-7352aa0bd399 nodeName:}" failed. No retries permitted until 2026-05-28 15:13:28.653598148 +0000 UTC m=+160.952667639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399") : configmap references non-existent config key: ca-bundle.crt
May 28 15:13:28.154171 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.154063    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.154337 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.154310    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.155359 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.154963    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.155359 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.155336    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.155513 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.155493    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.155621 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.155599    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.155673 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.155640    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.155988 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.155967    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-web-config\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.156053 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.156002    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.156374 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.156357    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-out\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.156611 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.156596    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.162561 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.162538    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75rm\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-kube-api-access-r75rm\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.454902 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.454817    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:13:28.457577 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.457554    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/713bf691-c64b-485f-b8f8-7094daac71eb-metrics-tls\") pod \"dns-default-9n5pb\" (UID: \"713bf691-c64b-485f-b8f8-7094daac71eb\") " pod="openshift-dns/dns-default-9n5pb"
May 28 15:13:28.555379 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.555346    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:13:28.557681 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.557660    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37fbb52e-9da0-488d-b0a2-8706e14120a8-cert\") pod \"ingress-canary-5lwgs\" (UID: \"37fbb52e-9da0-488d-b0a2-8706e14120a8\") " pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:13:28.598734 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.598704    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8f6nh\""
May 28 15:13:28.606822 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.606795    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9n5pb"
May 28 15:13:28.656258 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.656221    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.657783 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.657360    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:28.736997 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.736920    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9n5pb"]
May 28 15:13:28.811735 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.811702    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkd7p" event={"ID":"a131cb29-6526-475f-b894-1bcedb93b59a","Type":"ContainerStarted","Data":"d4d9818d2980b2f0ca3748f5fc00201c458532ab17f827dc6824e90c2dc9c9ff"}
May 28 15:13:28.858615 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:28.858581    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713bf691_c64b_485f_b8f8_7094daac71eb.slice/crio-764e0a203747178c1e9c5485feeab5ee5008378e424d9ac198c95b2102d06d20 WatchSource:0}: Error finding container 764e0a203747178c1e9c5485feeab5ee5008378e424d9ac198c95b2102d06d20: Status 404 returned error can't find the container with id 764e0a203747178c1e9c5485feeab5ee5008378e424d9ac198c95b2102d06d20
May 28 15:13:28.878270 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:28.878243    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 15:13:29.028333 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.028247    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 15:13:29.030872 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:29.030841    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9422d2e_f331_4f5d_943a_7352aa0bd399.slice/crio-8b9ad4af1c574964ccf7aec114cc24207452a0352a91c39817dd3f4b844284b8 WatchSource:0}: Error finding container 8b9ad4af1c574964ccf7aec114cc24207452a0352a91c39817dd3f4b844284b8: Status 404 returned error can't find the container with id 8b9ad4af1c574964ccf7aec114cc24207452a0352a91c39817dd3f4b844284b8
May 28 15:13:29.815792 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.815687    2582 generic.go:358] "Generic (PLEG): container finished" podID="a131cb29-6526-475f-b894-1bcedb93b59a" containerID="bb83f27df90571ec1faeaaa3f68275b449e75d94128afda7524ab76200611e35" exitCode=0
May 28 15:13:29.815792 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.815737    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkd7p" event={"ID":"a131cb29-6526-475f-b894-1bcedb93b59a","Type":"ContainerDied","Data":"bb83f27df90571ec1faeaaa3f68275b449e75d94128afda7524ab76200611e35"}
May 28 15:13:29.817019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.816986    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerStarted","Data":"8b9ad4af1c574964ccf7aec114cc24207452a0352a91c39817dd3f4b844284b8"}
May 28 15:13:29.820228 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.818028    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9n5pb" event={"ID":"713bf691-c64b-485f-b8f8-7094daac71eb","Type":"ContainerStarted","Data":"764e0a203747178c1e9c5485feeab5ee5008378e424d9ac198c95b2102d06d20"}
May 28 15:13:29.858088 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.858057    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"]
May 28 15:13:29.862702 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.862679    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:29.865714 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.865560    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 28 15:13:29.865714 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.865577    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 28 15:13:29.865714 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.865593    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 28 15:13:29.865714 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.865635    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d46rhukeh1kpd\""
May 28 15:13:29.865714 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.865582    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 28 15:13:29.866036 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.865932    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5x6tv\""
May 28 15:13:29.866036 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.865992    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 28 15:13:29.873814 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.873562    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"]
May 28 15:13:29.968973 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.968943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:29.969152 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.968989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:29.969152 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.969062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-grpc-tls\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:29.969152 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.969100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqbwx\" (UniqueName: \"kubernetes.io/projected/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-kube-api-access-lqbwx\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:29.969317 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.969154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:29.969317 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.969181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:29.969317 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.969273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-tls\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:29.969317 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:29.969313    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-metrics-client-ca\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.071191 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.070154    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.071191 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.070206    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.071191 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.070259    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-tls\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.071191 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.070287    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-metrics-client-ca\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.071191 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.070321    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.071191 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.070351    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.071191 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.070421    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-grpc-tls\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.071191 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.070469    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbwx\" (UniqueName: \"kubernetes.io/projected/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-kube-api-access-lqbwx\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.072076 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.072030    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-metrics-client-ca\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.074695 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.074663    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-grpc-tls\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.076922 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.076900    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.077018 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.076965    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.078720 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.078678    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.078955 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.078935    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-tls\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.079510 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.079468    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbwx\" (UniqueName: \"kubernetes.io/projected/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-kube-api-access-lqbwx\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.079981 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.079944    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3c8ad83a-578d-4851-bbdf-d839d3f2ef34-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fb8859d8-bbq8s\" (UID: \"3c8ad83a-578d-4851-bbdf-d839d3f2ef34\") " pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.182832 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.182790    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:30.797285 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.796662    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"]
May 28 15:13:30.817406 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:30.817376    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8ad83a_578d_4851_bbdf_d839d3f2ef34.slice/crio-bd914c3fb5ee47085a36101f89069868b0f28db40ceff253317f310b5f14cdd1 WatchSource:0}: Error finding container bd914c3fb5ee47085a36101f89069868b0f28db40ceff253317f310b5f14cdd1: Status 404 returned error can't find the container with id bd914c3fb5ee47085a36101f89069868b0f28db40ceff253317f310b5f14cdd1
May 28 15:13:30.821452 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.821424    2582 generic.go:358] "Generic (PLEG): container finished" podID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerID="66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940" exitCode=0
May 28 15:13:30.821556 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.821464    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerDied","Data":"66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940"}
May 28 15:13:30.822990 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.822966    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9n5pb" event={"ID":"713bf691-c64b-485f-b8f8-7094daac71eb","Type":"ContainerStarted","Data":"a736b8899b1b0ab5087c620d2f47f12797a7b7a9387aff5783af5da3439baf7b"}
May 28 15:13:30.824007 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.823983    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s" event={"ID":"3c8ad83a-578d-4851-bbdf-d839d3f2ef34","Type":"ContainerStarted","Data":"bd914c3fb5ee47085a36101f89069868b0f28db40ceff253317f310b5f14cdd1"}
May 28 15:13:30.825636 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.825612    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkd7p" event={"ID":"a131cb29-6526-475f-b894-1bcedb93b59a","Type":"ContainerStarted","Data":"0a139d0ad1ed8378ffef07140cb15ead852bb464ef74eadc1652fb0099f937a0"}
May 28 15:13:30.825715 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.825641    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkd7p" event={"ID":"a131cb29-6526-475f-b894-1bcedb93b59a","Type":"ContainerStarted","Data":"3bc4f50c207b17bf0f14bf77de1b6112ba8571eef3bcb6c3ccf0800c25ee12be"}
May 28 15:13:30.876458 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:30.876375    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kkd7p" podStartSLOduration=3.793228138 podStartE2EDuration="4.876357273s" podCreationTimestamp="2026-05-28 15:13:26 +0000 UTC" firstStartedPulling="2026-05-28 15:13:27.81998687 +0000 UTC m=+160.119056376" lastFinishedPulling="2026-05-28 15:13:28.903116015 +0000 UTC m=+161.202185511" observedRunningTime="2026-05-28 15:13:30.87566736 +0000 UTC m=+163.174736885" watchObservedRunningTime="2026-05-28 15:13:30.876357273 +0000 UTC m=+163.175426784"
May 28 15:13:31.307336 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.307296    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"]
May 28 15:13:31.310459 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.310440    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.313715 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.313692    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 28 15:13:31.313797 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.313698    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 28 15:13:31.314271 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.314251    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 28 15:13:31.314342 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.314258    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pg2bf\""
May 28 15:13:31.315358 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.315337    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9hqc4hp1qqrco\""
May 28 15:13:31.321665 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.321645    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 28 15:13:31.334458 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.334434    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"]
May 28 15:13:31.382169 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.382129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxrjc\" (UniqueName: \"kubernetes.io/projected/64bef1cf-1067-4750-a373-48a51752bc00-kube-api-access-qxrjc\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.382356 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.382191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/64bef1cf-1067-4750-a373-48a51752bc00-audit-log\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.382356 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.382246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/64bef1cf-1067-4750-a373-48a51752bc00-secret-metrics-server-tls\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.382471 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.382363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/64bef1cf-1067-4750-a373-48a51752bc00-metrics-server-audit-profiles\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.382471 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.382421    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64bef1cf-1067-4750-a373-48a51752bc00-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.382572 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.382532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/64bef1cf-1067-4750-a373-48a51752bc00-client-ca-bundle\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.382625 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.382599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/64bef1cf-1067-4750-a373-48a51752bc00-secret-metrics-server-client-certs\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.483580 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.483514    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qxrjc\" (UniqueName: \"kubernetes.io/projected/64bef1cf-1067-4750-a373-48a51752bc00-kube-api-access-qxrjc\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.483762 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.483628    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/64bef1cf-1067-4750-a373-48a51752bc00-audit-log\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.483762 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.483672    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/64bef1cf-1067-4750-a373-48a51752bc00-secret-metrics-server-tls\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.483762 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.483703    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/64bef1cf-1067-4750-a373-48a51752bc00-metrics-server-audit-profiles\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.483762 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.483757    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64bef1cf-1067-4750-a373-48a51752bc00-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.483955 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.483811    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/64bef1cf-1067-4750-a373-48a51752bc00-client-ca-bundle\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.483955 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.483878    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/64bef1cf-1067-4750-a373-48a51752bc00-secret-metrics-server-client-certs\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.484218 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.484169    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/64bef1cf-1067-4750-a373-48a51752bc00-audit-log\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.484715 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.484665    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64bef1cf-1067-4750-a373-48a51752bc00-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.484893 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.484849    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/64bef1cf-1067-4750-a373-48a51752bc00-metrics-server-audit-profiles\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.487110 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.487091    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/64bef1cf-1067-4750-a373-48a51752bc00-secret-metrics-server-client-certs\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.487224 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.487204    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/64bef1cf-1067-4750-a373-48a51752bc00-client-ca-bundle\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.487290 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.487238    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/64bef1cf-1067-4750-a373-48a51752bc00-secret-metrics-server-tls\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.491884 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.491859    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxrjc\" (UniqueName: \"kubernetes.io/projected/64bef1cf-1067-4750-a373-48a51752bc00-kube-api-access-qxrjc\") pod \"metrics-server-b99c6c9bf-l6sp2\" (UID: \"64bef1cf-1067-4750-a373-48a51752bc00\") " pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.619857 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.619744    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:31.766350 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.766113    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"]
May 28 15:13:31.769477 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:31.769443    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bef1cf_1067_4750_a373_48a51752bc00.slice/crio-96201c0c9880865244374e829c95c699ed4035a6585a6cf9914a7e843806d798 WatchSource:0}: Error finding container 96201c0c9880865244374e829c95c699ed4035a6585a6cf9914a7e843806d798: Status 404 returned error can't find the container with id 96201c0c9880865244374e829c95c699ed4035a6585a6cf9914a7e843806d798
May 28 15:13:31.831011 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.830959    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9n5pb" event={"ID":"713bf691-c64b-485f-b8f8-7094daac71eb","Type":"ContainerStarted","Data":"ddcc29522260ad668ccf9de9154ed33af190e6cfc65706da389be2e36da0e2f1"}
May 28 15:13:31.831480 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.831216    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9n5pb"
May 28 15:13:31.832759 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.832720    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2" event={"ID":"64bef1cf-1067-4750-a373-48a51752bc00","Type":"ContainerStarted","Data":"96201c0c9880865244374e829c95c699ed4035a6585a6cf9914a7e843806d798"}
May 28 15:13:31.849519 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:31.849458    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-9n5pb" podStartSLOduration=130.051720057 podStartE2EDuration="2m11.84944231s" podCreationTimestamp="2026-05-28 15:11:20 +0000 UTC" firstStartedPulling="2026-05-28 15:13:28.860556113 +0000 UTC m=+161.159625604" lastFinishedPulling="2026-05-28 15:13:30.65827836 +0000 UTC m=+162.957347857" observedRunningTime="2026-05-28 15:13:31.848807973 +0000 UTC m=+164.147877485" watchObservedRunningTime="2026-05-28 15:13:31.84944231 +0000 UTC m=+164.148511875"
May 28 15:13:33.068247 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.068208    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 15:13:33.073344 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.073316    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.076874 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.076682    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 28 15:13:33.076874 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.076872    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 28 15:13:33.077053 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.076817    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 28 15:13:33.077053 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.076829    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 28 15:13:33.077168 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.076860    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 28 15:13:33.077168 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.076738    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d7nbj\""
May 28 15:13:33.077895 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.077876    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 28 15:13:33.078371 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.078210    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 28 15:13:33.078371 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.078240    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 28 15:13:33.078371 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.078213    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 28 15:13:33.078371 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.078323    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a8dcglaejdj8r\""
May 28 15:13:33.078653 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.078539    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 28 15:13:33.078753 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.078718    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 28 15:13:33.080926 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.080905    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 28 15:13:33.086740 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.085801    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 15:13:33.100200 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100200 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100145    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100200 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100436 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100436 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100436 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100436 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100290    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7gp4\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-kube-api-access-f7gp4\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100436 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-web-config\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100436 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100436 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config-out\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.100790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.100707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201469    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201509    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201538    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201582    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201607    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201639    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201667    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7gp4\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-kube-api-access-f7gp4\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201705    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-web-config\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201729    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201757    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config-out\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201809    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201851    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201881    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201918    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201945    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.201998    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.202547 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.202057    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.203446 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.202084    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.203446 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.202167    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.212146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.209065    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.212146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.209664    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.212146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.211664    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.212871 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.212843    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.221019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.220991    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.222700 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.222514    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.223273 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.223166    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config-out\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.223495 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.223477    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.223881 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.221936    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.224470 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.224431    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-web-config\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.225410 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.225385    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.226044 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.225591    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.226566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.226233    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.226566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.226509    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.226702 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.226631    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.227038 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.226985    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.228733 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.228594    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7gp4\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-kube-api-access-f7gp4\") pod \"prometheus-k8s-0\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.390069 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.390006    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:33.842585 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.842556    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerStarted","Data":"7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846"}
May 28 15:13:33.842714 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.842676    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerStarted","Data":"f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758"}
May 28 15:13:33.842714 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.842696    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerStarted","Data":"d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60"}
May 28 15:13:33.842814 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.842708    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerStarted","Data":"51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a"}
May 28 15:13:33.844417 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.844391    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s" event={"ID":"3c8ad83a-578d-4851-bbdf-d839d3f2ef34","Type":"ContainerStarted","Data":"b6b0054ab2fc28c645defd9d190a2cbec223ed2646d412c229b435c9b9dd9ee8"}
May 28 15:13:33.844417 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.844418    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s" event={"ID":"3c8ad83a-578d-4851-bbdf-d839d3f2ef34","Type":"ContainerStarted","Data":"043a9ad4dc9001f0b16c861e631835f507ffaa3f56f2e224cfb758674c0ced5b"}
May 28 15:13:33.844549 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.844427    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s" event={"ID":"3c8ad83a-578d-4851-bbdf-d839d3f2ef34","Type":"ContainerStarted","Data":"4aed05a2114e7f9e77be4ac4e29146177bf34be7baebcbbf5688e23f8fc642b3"}
May 28 15:13:33.880614 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:33.880162    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 15:13:33.884541 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:33.884502    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04e72f0_ca2e_4ebc_b795_f7a98afa3a22.slice/crio-3a55fea68e99cf7347b04ae07c72ef14ac18ad302c3327111d46a0c891d38997 WatchSource:0}: Error finding container 3a55fea68e99cf7347b04ae07c72ef14ac18ad302c3327111d46a0c891d38997: Status 404 returned error can't find the container with id 3a55fea68e99cf7347b04ae07c72ef14ac18ad302c3327111d46a0c891d38997
May 28 15:13:34.850475 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.850386    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s" event={"ID":"3c8ad83a-578d-4851-bbdf-d839d3f2ef34","Type":"ContainerStarted","Data":"03a0d13ca700da57012c5022a240a3c6654e315d7d73de37b3a3e3b3ccc29a00"}
May 28 15:13:34.850475 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.850429    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s" event={"ID":"3c8ad83a-578d-4851-bbdf-d839d3f2ef34","Type":"ContainerStarted","Data":"f9bb595b93a5e15b6e745b0bed97e3e03a6be8249cce7cdb09f6a32e18a8b672"}
May 28 15:13:34.850475 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.850443    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s" event={"ID":"3c8ad83a-578d-4851-bbdf-d839d3f2ef34","Type":"ContainerStarted","Data":"80e922601bde60d57942739ace5db55fd357eee4147c50c249db7d98efa7e000"}
May 28 15:13:34.850957 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.850663    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:34.851675 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.851655    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2" event={"ID":"64bef1cf-1067-4750-a373-48a51752bc00","Type":"ContainerStarted","Data":"2b02d67b6b77689c61e8436f5e375e6586554d8ba989b12f0b923d437c0fd4d3"}
May 28 15:13:34.852783 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.852758    2582 generic.go:358] "Generic (PLEG): container finished" podID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerID="c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea" exitCode=0
May 28 15:13:34.852867 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.852835    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerDied","Data":"c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea"}
May 28 15:13:34.852934 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.852867    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerStarted","Data":"3a55fea68e99cf7347b04ae07c72ef14ac18ad302c3327111d46a0c891d38997"}
May 28 15:13:34.855691 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.855673    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerStarted","Data":"356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2"}
May 28 15:13:34.855757 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.855698    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerStarted","Data":"3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb"}
May 28 15:13:34.905533 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.905470    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.531131801 podStartE2EDuration="7.905455974s" podCreationTimestamp="2026-05-28 15:13:27 +0000 UTC" firstStartedPulling="2026-05-28 15:13:29.032646845 +0000 UTC m=+161.331716335" lastFinishedPulling="2026-05-28 15:13:34.406971012 +0000 UTC m=+166.706040508" observedRunningTime="2026-05-28 15:13:34.904445223 +0000 UTC m=+167.203514726" watchObservedRunningTime="2026-05-28 15:13:34.905455974 +0000 UTC m=+167.204525484"
May 28 15:13:34.905711 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.905635    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s" podStartSLOduration=2.31955553 podStartE2EDuration="5.905629345s" podCreationTimestamp="2026-05-28 15:13:29 +0000 UTC" firstStartedPulling="2026-05-28 15:13:30.819345214 +0000 UTC m=+163.118414704" lastFinishedPulling="2026-05-28 15:13:34.405419026 +0000 UTC m=+166.704488519" observedRunningTime="2026-05-28 15:13:34.875132856 +0000 UTC m=+167.174202357" watchObservedRunningTime="2026-05-28 15:13:34.905629345 +0000 UTC m=+167.204698856"
May 28 15:13:34.951337 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:34.951286    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2" podStartSLOduration=1.939330652 podStartE2EDuration="3.951273724s" podCreationTimestamp="2026-05-28 15:13:31 +0000 UTC" firstStartedPulling="2026-05-28 15:13:31.77225381 +0000 UTC m=+164.071323311" lastFinishedPulling="2026-05-28 15:13:33.784196883 +0000 UTC m=+166.083266383" observedRunningTime="2026-05-28 15:13:34.950066515 +0000 UTC m=+167.249136009" watchObservedRunningTime="2026-05-28 15:13:34.951273724 +0000 UTC m=+167.250343241"
May 28 15:13:35.287790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:35.287757    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:13:35.291011 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:35.290991    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ljmrm\""
May 28 15:13:35.298542 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:35.298520    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5lwgs"
May 28 15:13:35.299680 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:35.299664    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:13:35.424081 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:35.423991    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5lwgs"]
May 28 15:13:35.426456 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:13:35.426424    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fbb52e_9da0_488d_b0a2_8706e14120a8.slice/crio-37f26d92ab7b38107dbeac8480a2b192916b687fa4d66defbb107aa340600d28 WatchSource:0}: Error finding container 37f26d92ab7b38107dbeac8480a2b192916b687fa4d66defbb107aa340600d28: Status 404 returned error can't find the container with id 37f26d92ab7b38107dbeac8480a2b192916b687fa4d66defbb107aa340600d28
May 28 15:13:35.860028 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:35.859991    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5lwgs" event={"ID":"37fbb52e-9da0-488d-b0a2-8706e14120a8","Type":"ContainerStarted","Data":"37f26d92ab7b38107dbeac8480a2b192916b687fa4d66defbb107aa340600d28"}
May 28 15:13:36.288424 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:36.288336    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:13:38.873053 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:38.873021    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerStarted","Data":"539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08"}
May 28 15:13:38.873053 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:38.873056    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerStarted","Data":"935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c"}
May 28 15:13:38.873667 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:38.873072    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerStarted","Data":"4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f"}
May 28 15:13:38.873667 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:38.873084    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerStarted","Data":"c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4"}
May 28 15:13:38.873667 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:38.873092    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerStarted","Data":"ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6"}
May 28 15:13:38.873667 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:38.873102    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerStarted","Data":"5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39"}
May 28 15:13:38.874301 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:38.874282    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5lwgs" event={"ID":"37fbb52e-9da0-488d-b0a2-8706e14120a8","Type":"ContainerStarted","Data":"a1d629e3242c8c9fd2f11a34a1af003620dd0e6704ededcf85a87ed0562bf253"}
May 28 15:13:38.900198 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:38.900151    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.322400796 podStartE2EDuration="5.900137147s" podCreationTimestamp="2026-05-28 15:13:33 +0000 UTC" firstStartedPulling="2026-05-28 15:13:34.853855795 +0000 UTC m=+167.152925287" lastFinishedPulling="2026-05-28 15:13:38.431592143 +0000 UTC m=+170.730661638" observedRunningTime="2026-05-28 15:13:38.897560739 +0000 UTC m=+171.196630306" watchObservedRunningTime="2026-05-28 15:13:38.900137147 +0000 UTC m=+171.199206652"
May 28 15:13:38.914548 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:38.914498    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5lwgs" podStartSLOduration=135.91546677 podStartE2EDuration="2m18.914484251s" podCreationTimestamp="2026-05-28 15:11:20 +0000 UTC" firstStartedPulling="2026-05-28 15:13:35.42832464 +0000 UTC m=+167.727394129" lastFinishedPulling="2026-05-28 15:13:38.427342114 +0000 UTC m=+170.726411610" observedRunningTime="2026-05-28 15:13:38.914200294 +0000 UTC m=+171.213269804" watchObservedRunningTime="2026-05-28 15:13:38.914484251 +0000 UTC m=+171.213553761"
May 28 15:13:39.288168 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:39.288112    2582 scope.go:117] "RemoveContainer" containerID="a00f5787af3b50ad9ef81446de3c14182bbb984ea1d1aea78da41205efe0470f"
May 28 15:13:39.878799 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:39.878772    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:13:39.879221 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:39.878892    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" event={"ID":"d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d","Type":"ContainerStarted","Data":"540a418d416588e28daff69d7efde97d903d2a7727961bd661073c5fe43a690e"}
May 28 15:13:39.879624 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:39.879599    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:13:39.887668 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:39.887645    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4"
May 28 15:13:39.908738 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:39.908682    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-k6lc4" podStartSLOduration=54.008182336 podStartE2EDuration="55.90866329s" podCreationTimestamp="2026-05-28 15:12:44 +0000 UTC" firstStartedPulling="2026-05-28 15:12:45.071746732 +0000 UTC m=+117.370816221" lastFinishedPulling="2026-05-28 15:12:46.972227673 +0000 UTC m=+119.271297175" observedRunningTime="2026-05-28 15:13:39.908302681 +0000 UTC m=+172.207372192" watchObservedRunningTime="2026-05-28 15:13:39.90866329 +0000 UTC m=+172.207732821"
May 28 15:13:40.313518 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.313424    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s" podUID="116f4a3b-549f-4ac0-97ad-b0269f62230d" containerName="registry" containerID="cri-o://12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b" gracePeriod=30
May 28 15:13:40.565351 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.565290    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:13:40.673758 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.673720    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/116f4a3b-549f-4ac0-97ad-b0269f62230d-ca-trust-extracted\") pod \"116f4a3b-549f-4ac0-97ad-b0269f62230d\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") "
May 28 15:13:40.673939 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.673769    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-image-registry-private-configuration\") pod \"116f4a3b-549f-4ac0-97ad-b0269f62230d\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") "
May 28 15:13:40.673939 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.673790    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmr5b\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-kube-api-access-xmr5b\") pod \"116f4a3b-549f-4ac0-97ad-b0269f62230d\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") "
May 28 15:13:40.673939 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.673807    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") pod \"116f4a3b-549f-4ac0-97ad-b0269f62230d\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") "
May 28 15:13:40.673939 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.673859    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-installation-pull-secrets\") pod \"116f4a3b-549f-4ac0-97ad-b0269f62230d\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") "
May 28 15:13:40.673939 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.673879    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-bound-sa-token\") pod \"116f4a3b-549f-4ac0-97ad-b0269f62230d\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") "
May 28 15:13:40.673939 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.673912    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-trusted-ca\") pod \"116f4a3b-549f-4ac0-97ad-b0269f62230d\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") "
May 28 15:13:40.674248 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.674030    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-certificates\") pod \"116f4a3b-549f-4ac0-97ad-b0269f62230d\" (UID: \"116f4a3b-549f-4ac0-97ad-b0269f62230d\") "
May 28 15:13:40.674689 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.674660    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "116f4a3b-549f-4ac0-97ad-b0269f62230d" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:13:40.674791 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.674664    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "116f4a3b-549f-4ac0-97ad-b0269f62230d" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:13:40.676305 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.676273    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-kube-api-access-xmr5b" (OuterVolumeSpecName: "kube-api-access-xmr5b") pod "116f4a3b-549f-4ac0-97ad-b0269f62230d" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d"). InnerVolumeSpecName "kube-api-access-xmr5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:13:40.676409 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.676315    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "116f4a3b-549f-4ac0-97ad-b0269f62230d" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:13:40.676409 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.676331    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "116f4a3b-549f-4ac0-97ad-b0269f62230d" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:13:40.676409 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.676380    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "116f4a3b-549f-4ac0-97ad-b0269f62230d" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:13:40.676740 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.676720    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "116f4a3b-549f-4ac0-97ad-b0269f62230d" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:13:40.682385 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.682364    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116f4a3b-549f-4ac0-97ad-b0269f62230d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "116f4a3b-549f-4ac0-97ad-b0269f62230d" (UID: "116f4a3b-549f-4ac0-97ad-b0269f62230d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:13:40.775397 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.775362    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-installation-pull-secrets\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:13:40.775397 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.775391    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-bound-sa-token\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:13:40.775397 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.775401    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-trusted-ca\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:13:40.775397 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.775409    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-certificates\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:13:40.775639 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.775418    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/116f4a3b-549f-4ac0-97ad-b0269f62230d-ca-trust-extracted\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:13:40.775639 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.775428    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/116f4a3b-549f-4ac0-97ad-b0269f62230d-image-registry-private-configuration\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:13:40.775639 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.775439    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmr5b\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-kube-api-access-xmr5b\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:13:40.775639 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.775448    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/116f4a3b-549f-4ac0-97ad-b0269f62230d-registry-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:13:40.865685 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.865661    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5fb8859d8-bbq8s"
May 28 15:13:40.883296 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.883266    2582 generic.go:358] "Generic (PLEG): container finished" podID="116f4a3b-549f-4ac0-97ad-b0269f62230d" containerID="12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b" exitCode=0
May 28 15:13:40.883632 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.883327    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s"
May 28 15:13:40.883632 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.883356    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s" event={"ID":"116f4a3b-549f-4ac0-97ad-b0269f62230d","Type":"ContainerDied","Data":"12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b"}
May 28 15:13:40.883632 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.883391    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5645c46bf9-mxj7s" event={"ID":"116f4a3b-549f-4ac0-97ad-b0269f62230d","Type":"ContainerDied","Data":"3e6a1f53cd6234dd718ad49c1b9dd0456015e8863a5ff2974b69d4d0a58b9e15"}
May 28 15:13:40.883632 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.883406    2582 scope.go:117] "RemoveContainer" containerID="12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b"
May 28 15:13:40.891164 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.891114    2582 scope.go:117] "RemoveContainer" containerID="12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b"
May 28 15:13:40.891434 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:13:40.891411    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b\": container with ID starting with 12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b not found: ID does not exist" containerID="12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b"
May 28 15:13:40.891520 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.891446    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b"} err="failed to get container status \"12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b\": rpc error: code = NotFound desc = could not find container \"12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b\": container with ID starting with 12d36147c25ecfc7ee6a2926169bba62b3b7ce7092606852271693efa4cf0b5b not found: ID does not exist"
May 28 15:13:40.919256 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.919227    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5645c46bf9-mxj7s"]
May 28 15:13:40.922541 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:40.922517    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5645c46bf9-mxj7s"]
May 28 15:13:41.839052 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:41.839026    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9n5pb"
May 28 15:13:42.292954 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:42.292924    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="116f4a3b-549f-4ac0-97ad-b0269f62230d" path="/var/lib/kubelet/pods/116f4a3b-549f-4ac0-97ad-b0269f62230d/volumes"
May 28 15:13:43.390490 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:43.390455    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:13:51.620762 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:51.620727    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:13:51.621224 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:13:51.620798    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:14:11.625796 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:11.625765    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:14:11.629576 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:11.629551    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b99c6c9bf-l6sp2"
May 28 15:14:33.392032 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:33.390600    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:33.418262 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:33.418229    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:34.051083 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:34.050794    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:47.142143 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:47.142081    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 15:14:47.142599 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:47.142572    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="alertmanager" containerID="cri-o://51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a" gracePeriod=120
May 28 15:14:47.142705 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:47.142631    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy-metric" containerID="cri-o://3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb" gracePeriod=120
May 28 15:14:47.142843 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:47.142688    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy" containerID="cri-o://7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846" gracePeriod=120
May 28 15:14:47.142843 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:47.142634    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy-web" containerID="cri-o://f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758" gracePeriod=120
May 28 15:14:47.142843 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:47.142691    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="prom-label-proxy" containerID="cri-o://356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2" gracePeriod=120
May 28 15:14:47.142843 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:47.142652    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="config-reloader" containerID="cri-o://d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60" gracePeriod=120
May 28 15:14:48.077250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.077211    2582 generic.go:358] "Generic (PLEG): container finished" podID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerID="356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2" exitCode=0
May 28 15:14:48.077250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.077245    2582 generic.go:358] "Generic (PLEG): container finished" podID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerID="7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846" exitCode=0
May 28 15:14:48.077250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.077253    2582 generic.go:358] "Generic (PLEG): container finished" podID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerID="d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60" exitCode=0
May 28 15:14:48.077473 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.077262    2582 generic.go:358] "Generic (PLEG): container finished" podID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerID="51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a" exitCode=0
May 28 15:14:48.077473 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.077277    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerDied","Data":"356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2"}
May 28 15:14:48.077473 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.077313    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerDied","Data":"7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846"}
May 28 15:14:48.077473 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.077324    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerDied","Data":"d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60"}
May 28 15:14:48.077473 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.077332    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerDied","Data":"51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a"}
May 28 15:14:48.384601 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.384576    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:48.488490 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.488453    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-main-tls\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.488668 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.488503    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-volume\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.488668 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.488535    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-main-db\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.488668 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.488635    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-cluster-tls-config\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.488826 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.488678    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.488826 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.488711    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-web\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.488826 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.488742    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-trusted-ca-bundle\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.489010 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.488991    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r75rm\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-kube-api-access-r75rm\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.489078 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489047    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-out\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.489159 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489079    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-tls-assets\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.489159 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489096    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:14:48.489159 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489111    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-web-config\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.489310 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489169    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.489310 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489203    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-metrics-client-ca\") pod \"c9422d2e-f331-4f5d-943a-7352aa0bd399\" (UID: \"c9422d2e-f331-4f5d-943a-7352aa0bd399\") "
May 28 15:14:48.489310 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489230    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:14:48.489944 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489515    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-main-db\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.489944 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489540    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.489944 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.489908    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:14:48.491333 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.491307    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:48.491756 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.491730    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:48.492282 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.492260    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-volume" (OuterVolumeSpecName: "config-volume") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:48.492371 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.492258    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:14:48.492469 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.492440    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:48.492866 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.492844    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-kube-api-access-r75rm" (OuterVolumeSpecName: "kube-api-access-r75rm") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "kube-api-access-r75rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:14:48.492965 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.492935    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:48.493269 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.493252    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-out" (OuterVolumeSpecName: "config-out") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:14:48.495722 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.495703    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:48.501967 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.501947    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-web-config" (OuterVolumeSpecName: "web-config") pod "c9422d2e-f331-4f5d-943a-7352aa0bd399" (UID: "c9422d2e-f331-4f5d-943a-7352aa0bd399"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:48.590115 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590062    2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-volume\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590115 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590108    2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-cluster-tls-config\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590115 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590140    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590115 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590150    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590159    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r75rm\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-kube-api-access-r75rm\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590169    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9422d2e-f331-4f5d-943a-7352aa0bd399-config-out\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590177    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9422d2e-f331-4f5d-943a-7352aa0bd399-tls-assets\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590186    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-web-config\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590195    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590203    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9422d2e-f331-4f5d-943a-7352aa0bd399-metrics-client-ca\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:48.590402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:48.590211    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9422d2e-f331-4f5d-943a-7352aa0bd399-secret-alertmanager-main-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:49.083901 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.083866    2582 generic.go:358] "Generic (PLEG): container finished" podID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerID="3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb" exitCode=0
May 28 15:14:49.083901 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.083892    2582 generic.go:358] "Generic (PLEG): container finished" podID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerID="f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758" exitCode=0
May 28 15:14:49.084086 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.083951    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerDied","Data":"3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb"}
May 28 15:14:49.084086 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.083980    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.084086 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.083989    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerDied","Data":"f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758"}
May 28 15:14:49.084086 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.084002    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9422d2e-f331-4f5d-943a-7352aa0bd399","Type":"ContainerDied","Data":"8b9ad4af1c574964ccf7aec114cc24207452a0352a91c39817dd3f4b844284b8"}
May 28 15:14:49.084086 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.084016    2582 scope.go:117] "RemoveContainer" containerID="356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2"
May 28 15:14:49.091407 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.091300    2582 scope.go:117] "RemoveContainer" containerID="3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb"
May 28 15:14:49.097756 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.097738    2582 scope.go:117] "RemoveContainer" containerID="7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846"
May 28 15:14:49.103633 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.103615    2582 scope.go:117] "RemoveContainer" containerID="f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758"
May 28 15:14:49.108154 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.107928    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 15:14:49.112775 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.112756    2582 scope.go:117] "RemoveContainer" containerID="d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60"
May 28 15:14:49.113486 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.113465    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 15:14:49.119115 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.119096    2582 scope.go:117] "RemoveContainer" containerID="51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a"
May 28 15:14:49.125902 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.125876    2582 scope.go:117] "RemoveContainer" containerID="66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940"
May 28 15:14:49.131893 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.131878    2582 scope.go:117] "RemoveContainer" containerID="356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2"
May 28 15:14:49.132111 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:49.132094    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2\": container with ID starting with 356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2 not found: ID does not exist" containerID="356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2"
May 28 15:14:49.132175 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.132134    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2"} err="failed to get container status \"356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2\": rpc error: code = NotFound desc = could not find container \"356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2\": container with ID starting with 356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2 not found: ID does not exist"
May 28 15:14:49.132175 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.132154    2582 scope.go:117] "RemoveContainer" containerID="3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb"
May 28 15:14:49.132357 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:49.132338    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb\": container with ID starting with 3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb not found: ID does not exist" containerID="3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb"
May 28 15:14:49.132395 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.132363    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb"} err="failed to get container status \"3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb\": rpc error: code = NotFound desc = could not find container \"3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb\": container with ID starting with 3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb not found: ID does not exist"
May 28 15:14:49.132395 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.132380    2582 scope.go:117] "RemoveContainer" containerID="7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846"
May 28 15:14:49.132575 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:49.132558    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846\": container with ID starting with 7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846 not found: ID does not exist" containerID="7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846"
May 28 15:14:49.132612 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.132579    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846"} err="failed to get container status \"7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846\": rpc error: code = NotFound desc = could not find container \"7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846\": container with ID starting with 7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846 not found: ID does not exist"
May 28 15:14:49.132612 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.132592    2582 scope.go:117] "RemoveContainer" containerID="f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758"
May 28 15:14:49.132810 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:49.132794    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758\": container with ID starting with f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758 not found: ID does not exist" containerID="f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758"
May 28 15:14:49.132858 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.132812    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758"} err="failed to get container status \"f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758\": rpc error: code = NotFound desc = could not find container \"f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758\": container with ID starting with f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758 not found: ID does not exist"
May 28 15:14:49.132858 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.132824    2582 scope.go:117] "RemoveContainer" containerID="d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60"
May 28 15:14:49.133002 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:49.132989    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60\": container with ID starting with d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60 not found: ID does not exist" containerID="d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60"
May 28 15:14:49.133084 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.133005    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60"} err="failed to get container status \"d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60\": rpc error: code = NotFound desc = could not find container \"d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60\": container with ID starting with d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60 not found: ID does not exist"
May 28 15:14:49.133084 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.133016    2582 scope.go:117] "RemoveContainer" containerID="51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a"
May 28 15:14:49.133235 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:49.133213    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a\": container with ID starting with 51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a not found: ID does not exist" containerID="51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a"
May 28 15:14:49.133321 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.133238    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a"} err="failed to get container status \"51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a\": rpc error: code = NotFound desc = could not find container \"51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a\": container with ID starting with 51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a not found: ID does not exist"
May 28 15:14:49.133321 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.133250    2582 scope.go:117] "RemoveContainer" containerID="66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940"
May 28 15:14:49.133599 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:49.133573    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940\": container with ID starting with 66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940 not found: ID does not exist" containerID="66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940"
May 28 15:14:49.133706 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.133609    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940"} err="failed to get container status \"66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940\": rpc error: code = NotFound desc = could not find container \"66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940\": container with ID starting with 66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940 not found: ID does not exist"
May 28 15:14:49.133777 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.133646    2582 scope.go:117] "RemoveContainer" containerID="356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2"
May 28 15:14:49.134299 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.134271    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2"} err="failed to get container status \"356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2\": rpc error: code = NotFound desc = could not find container \"356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2\": container with ID starting with 356ec2e13bf05edb2f6470aaeafcdc6d2cd9dabccf44cb954905a66bd8d838d2 not found: ID does not exist"
May 28 15:14:49.134299 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.134295    2582 scope.go:117] "RemoveContainer" containerID="3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb"
May 28 15:14:49.134540 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.134520    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb"} err="failed to get container status \"3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb\": rpc error: code = NotFound desc = could not find container \"3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb\": container with ID starting with 3694a760ba35cd7a3c65900dedff428a9445da04a90ca2fb6c1d05b4af1ca1bb not found: ID does not exist"
May 28 15:14:49.134597 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.134542    2582 scope.go:117] "RemoveContainer" containerID="7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846"
May 28 15:14:49.134782 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.134766    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846"} err="failed to get container status \"7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846\": rpc error: code = NotFound desc = could not find container \"7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846\": container with ID starting with 7c6737e009389740f2ea1f987104bfd8c296be9dbeeab9f3cd37a320971f8846 not found: ID does not exist"
May 28 15:14:49.134835 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.134783    2582 scope.go:117] "RemoveContainer" containerID="f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758"
May 28 15:14:49.135040 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135019    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758"} err="failed to get container status \"f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758\": rpc error: code = NotFound desc = could not find container \"f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758\": container with ID starting with f386b446098eb982b953e5eebbec27c822d6380e2fa73ca0df8387ccf7997758 not found: ID does not exist"
May 28 15:14:49.135040 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135040    2582 scope.go:117] "RemoveContainer" containerID="d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60"
May 28 15:14:49.135263 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135246    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60"} err="failed to get container status \"d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60\": rpc error: code = NotFound desc = could not find container \"d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60\": container with ID starting with d12e76830f071e01b129abd2b74e7949a4252a84ed756d8b296aafad3fa57f60 not found: ID does not exist"
May 28 15:14:49.135323 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135272    2582 scope.go:117] "RemoveContainer" containerID="51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a"
May 28 15:14:49.135465 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135449    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 15:14:49.135501 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135466    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a"} err="failed to get container status \"51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a\": rpc error: code = NotFound desc = could not find container \"51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a\": container with ID starting with 51abd306933baa3352a99562428d04a497cf6eed0617180f5f4478bc7682f06a not found: ID does not exist"
May 28 15:14:49.135501 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135492    2582 scope.go:117] "RemoveContainer" containerID="66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940"
May 28 15:14:49.135707 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135684    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940"} err="failed to get container status \"66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940\": rpc error: code = NotFound desc = could not find container \"66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940\": container with ID starting with 66302fe43538c5e8da282a6481e9da64a865ffc6ed0c6552b56ac5e750015940 not found: ID does not exist"
May 28 15:14:49.135793 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135741    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="alertmanager"
May 28 15:14:49.135793 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135752    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="alertmanager"
May 28 15:14:49.135793 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135761    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="prom-label-proxy"
May 28 15:14:49.135793 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135766    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="prom-label-proxy"
May 28 15:14:49.135793 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135774    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116f4a3b-549f-4ac0-97ad-b0269f62230d" containerName="registry"
May 28 15:14:49.135793 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135779    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="116f4a3b-549f-4ac0-97ad-b0269f62230d" containerName="registry"
May 28 15:14:49.135793 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135784    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="config-reloader"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135800    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="config-reloader"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135806    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135811    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135822    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy-metric"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135827    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy-metric"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135832    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="init-config-reloader"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135837    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="init-config-reloader"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135846    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy-web"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135851    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy-web"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135908    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy-web"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135920    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="prom-label-proxy"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135926    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="alertmanager"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135933    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="116f4a3b-549f-4ac0-97ad-b0269f62230d" containerName="registry"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135939    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135945    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="config-reloader"
May 28 15:14:49.135998 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.135951    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" containerName="kube-rbac-proxy-metric"
May 28 15:14:49.141252 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.141235    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.144005 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.143944    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 28 15:14:49.144005 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.143971    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 28 15:14:49.144185 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.144151    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 28 15:14:49.144185 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.144181    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 28 15:14:49.144331 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.144207    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 28 15:14:49.144331 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.144207    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 28 15:14:49.144519 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.144482    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 28 15:14:49.144519 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.144513    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6vtss\""
May 28 15:14:49.144710 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.144697    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 28 15:14:49.149260 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.149244    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 28 15:14:49.151785 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.151762    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 15:14:49.195390 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195555 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195555 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195555 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7fhs\" (UniqueName: \"kubernetes.io/projected/9857b26e-d11e-42ed-ab8f-69828b9ca00b-kube-api-access-z7fhs\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195555 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9857b26e-d11e-42ed-ab8f-69828b9ca00b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195710 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-web-config\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195710 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9857b26e-d11e-42ed-ab8f-69828b9ca00b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195710 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9857b26e-d11e-42ed-ab8f-69828b9ca00b-config-out\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195806 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9857b26e-d11e-42ed-ab8f-69828b9ca00b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195806 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-config-volume\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195806 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195895 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9857b26e-d11e-42ed-ab8f-69828b9ca00b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.195895 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.195833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296492 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296450    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z7fhs\" (UniqueName: \"kubernetes.io/projected/9857b26e-d11e-42ed-ab8f-69828b9ca00b-kube-api-access-z7fhs\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296492 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296496    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9857b26e-d11e-42ed-ab8f-69828b9ca00b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296741 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296536    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-web-config\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296741 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296564    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9857b26e-d11e-42ed-ab8f-69828b9ca00b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296741 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296619    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9857b26e-d11e-42ed-ab8f-69828b9ca00b-config-out\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296741 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296647    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9857b26e-d11e-42ed-ab8f-69828b9ca00b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296741 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296675    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-config-volume\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296741 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296699    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296992 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296911    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9857b26e-d11e-42ed-ab8f-69828b9ca00b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.296992 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.296959    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.297090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.297001    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.297090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.297043    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.297090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.297073    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.297252 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.297209    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9857b26e-d11e-42ed-ab8f-69828b9ca00b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.297566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.297540    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9857b26e-d11e-42ed-ab8f-69828b9ca00b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.299765 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.299600    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-config-volume\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.299765 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.299648    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9857b26e-d11e-42ed-ab8f-69828b9ca00b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.299765 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.299693    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-web-config\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.299765 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.299705    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9857b26e-d11e-42ed-ab8f-69828b9ca00b-config-out\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.299765 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.299742    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.300080 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.300055    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.300262 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.300244    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9857b26e-d11e-42ed-ab8f-69828b9ca00b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.300337 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.300307    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.300454 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.300433    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.301572 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.301554    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9857b26e-d11e-42ed-ab8f-69828b9ca00b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.304594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.304574    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7fhs\" (UniqueName: \"kubernetes.io/projected/9857b26e-d11e-42ed-ab8f-69828b9ca00b-kube-api-access-z7fhs\") pod \"alertmanager-main-0\" (UID: \"9857b26e-d11e-42ed-ab8f-69828b9ca00b\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.450691 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.450657    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 15:14:49.578671 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:49.578639    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 15:14:49.581708 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:14:49.581682    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9857b26e_d11e_42ed_ab8f_69828b9ca00b.slice/crio-cc77230af01a63b0db74042058a86d0790084b1ac9c001b546c2193d8025f9b4 WatchSource:0}: Error finding container cc77230af01a63b0db74042058a86d0790084b1ac9c001b546c2193d8025f9b4: Status 404 returned error can't find the container with id cc77230af01a63b0db74042058a86d0790084b1ac9c001b546c2193d8025f9b4
May 28 15:14:50.088479 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:50.088448    2582 generic.go:358] "Generic (PLEG): container finished" podID="9857b26e-d11e-42ed-ab8f-69828b9ca00b" containerID="91875a9773fdf2523c6ae2517d8a8bd6222030d87bdf79dc1bd47b99e6745cb1" exitCode=0
May 28 15:14:50.088655 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:50.088531    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9857b26e-d11e-42ed-ab8f-69828b9ca00b","Type":"ContainerDied","Data":"91875a9773fdf2523c6ae2517d8a8bd6222030d87bdf79dc1bd47b99e6745cb1"}
May 28 15:14:50.088655 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:50.088564    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9857b26e-d11e-42ed-ab8f-69828b9ca00b","Type":"ContainerStarted","Data":"cc77230af01a63b0db74042058a86d0790084b1ac9c001b546c2193d8025f9b4"}
May 28 15:14:50.293911 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:50.293881    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9422d2e-f331-4f5d-943a-7352aa0bd399" path="/var/lib/kubelet/pods/c9422d2e-f331-4f5d-943a-7352aa0bd399/volumes"
May 28 15:14:51.100524 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.100483    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9857b26e-d11e-42ed-ab8f-69828b9ca00b","Type":"ContainerStarted","Data":"cf7f86ac452a8c1aff29aba387a738e71eddc9a15a4ea60f3a85a934443c07e8"}
May 28 15:14:51.100524 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.100517    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9857b26e-d11e-42ed-ab8f-69828b9ca00b","Type":"ContainerStarted","Data":"6305c096f3a40efcf01e90a990a441814126cc070d9d529ba0139171c7e4d5f6"}
May 28 15:14:51.100524 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.100526    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9857b26e-d11e-42ed-ab8f-69828b9ca00b","Type":"ContainerStarted","Data":"dfdc42c91a3f8b1780daffdbfc7bf9d19f84c9e1eb52f88c0fc789823b849f79"}
May 28 15:14:51.100524 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.100534    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9857b26e-d11e-42ed-ab8f-69828b9ca00b","Type":"ContainerStarted","Data":"7eaf91f9b7b55eebff7dc57399357ea80492b2f351e92c43ab636dca2b499b5e"}
May 28 15:14:51.101151 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.100542    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9857b26e-d11e-42ed-ab8f-69828b9ca00b","Type":"ContainerStarted","Data":"c73c31ec31fc06c69eea5ac3de79e5780745d24ff0fbdf81c29d2d03b8dcfe13"}
May 28 15:14:51.101151 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.100550    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9857b26e-d11e-42ed-ab8f-69828b9ca00b","Type":"ContainerStarted","Data":"cc3110a2e33d3bf720c220d5c6627d73e8409e096bb435eddd4faad050bc6abe"}
May 28 15:14:51.164528 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.164466    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.164448871 podStartE2EDuration="2.164448871s" podCreationTimestamp="2026-05-28 15:14:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:14:51.161501328 +0000 UTC m=+243.460570840" watchObservedRunningTime="2026-05-28 15:14:51.164448871 +0000 UTC m=+243.463518381"
May 28 15:14:51.398060 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.398027    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 15:14:51.398514 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.398479    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="prometheus" containerID="cri-o://5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39" gracePeriod=600
May 28 15:14:51.398514 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.398497    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy" containerID="cri-o://935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c" gracePeriod=600
May 28 15:14:51.398682 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.398526    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="thanos-sidecar" containerID="cri-o://c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4" gracePeriod=600
May 28 15:14:51.398682 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.398542    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy-web" containerID="cri-o://4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f" gracePeriod=600
May 28 15:14:51.398682 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.398547    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy-thanos" containerID="cri-o://539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08" gracePeriod=600
May 28 15:14:51.398682 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:51.398605    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="config-reloader" containerID="cri-o://ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6" gracePeriod=600
May 28 15:14:52.106169 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106136    2582 generic.go:358] "Generic (PLEG): container finished" podID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerID="539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08" exitCode=0
May 28 15:14:52.106169 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106148    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerDied","Data":"539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08"}
May 28 15:14:52.106169 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106161    2582 generic.go:358] "Generic (PLEG): container finished" podID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerID="935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c" exitCode=0
May 28 15:14:52.106169 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106168    2582 generic.go:358] "Generic (PLEG): container finished" podID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerID="c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4" exitCode=0
May 28 15:14:52.106169 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106174    2582 generic.go:358] "Generic (PLEG): container finished" podID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerID="ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6" exitCode=0
May 28 15:14:52.106169 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106179    2582 generic.go:358] "Generic (PLEG): container finished" podID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerID="5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39" exitCode=0
May 28 15:14:52.106169 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106180    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerDied","Data":"935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c"}
May 28 15:14:52.106699 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106191    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerDied","Data":"c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4"}
May 28 15:14:52.106699 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106201    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerDied","Data":"ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6"}
May 28 15:14:52.106699 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.106210    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerDied","Data":"5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39"}
May 28 15:14:52.649339 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.649317    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:52.728355 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728274    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-metrics-client-certs\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728355 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728332    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728355 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728356    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-web-config\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728583 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728518    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-thanos-prometheus-http-client-file\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728583 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728569    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728688 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728645    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-metrics-client-ca\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728743 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728689    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-tls\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728743 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728712    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-rulefiles-0\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728743 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728737    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728883 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728766    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-db\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728883 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728807    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config-out\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728883 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728834    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-tls-assets\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.728883 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728874    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-kubelet-serving-ca-bundle\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.729067 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728911    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-kube-rbac-proxy\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.729067 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728953    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7gp4\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-kube-api-access-f7gp4\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.729067 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.728983    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-trusted-ca-bundle\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.729067 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.729009    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-serving-certs-ca-bundle\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.729067 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.729037    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-grpc-tls\") pod \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\" (UID: \"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22\") "
May 28 15:14:52.729334 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.729308    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:14:52.730742 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.730712    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:14:52.730862 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.730754    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:52.731739 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.731011    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:52.731739 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.731062    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:52.731739 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.731690    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:14:52.731906 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.731794    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:14:52.731906 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.731870    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:14:52.732303 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.732236    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:14:52.732303 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.732273    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config" (OuterVolumeSpecName: "config") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:52.732425 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.732342    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:52.732548 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.732524    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:52.733035 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.733012    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config-out" (OuterVolumeSpecName: "config-out") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:14:52.733691 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.733669    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:14:52.734174 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.734149    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:52.734267 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.734167    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:52.734642 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.734623    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-kube-api-access-f7gp4" (OuterVolumeSpecName: "kube-api-access-f7gp4") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "kube-api-access-f7gp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:14:52.741814 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.741795    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-web-config" (OuterVolumeSpecName: "web-config") pod "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" (UID: "d04e72f0-ca2e-4ebc-b795-f7a98afa3a22"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:14:52.830362 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830323    2582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830362 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830355    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830362 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830368    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-metrics-client-ca\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830379    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830389    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830398    2582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830407    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-k8s-db\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830415    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-config-out\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830422    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-tls-assets\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830430    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830440    2582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-kube-rbac-proxy\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830448    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7gp4\" (UniqueName: \"kubernetes.io/projected/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-kube-api-access-f7gp4\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830458    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830467    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830475    2582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-grpc-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830484    2582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-metrics-client-certs\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830492    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:52.830563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:52.830502    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22-web-config\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:14:53.111549 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.111515    2582 generic.go:358] "Generic (PLEG): container finished" podID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerID="4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f" exitCode=0
May 28 15:14:53.111906 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.111593    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerDied","Data":"4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f"}
May 28 15:14:53.111906 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.111633    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d04e72f0-ca2e-4ebc-b795-f7a98afa3a22","Type":"ContainerDied","Data":"3a55fea68e99cf7347b04ae07c72ef14ac18ad302c3327111d46a0c891d38997"}
May 28 15:14:53.111906 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.111634    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.111906 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.111645    2582 scope.go:117] "RemoveContainer" containerID="539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08"
May 28 15:14:53.119223 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.119192    2582 scope.go:117] "RemoveContainer" containerID="935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c"
May 28 15:14:53.125866 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.125845    2582 scope.go:117] "RemoveContainer" containerID="4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f"
May 28 15:14:53.131850 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.131832    2582 scope.go:117] "RemoveContainer" containerID="c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4"
May 28 15:14:53.135236 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.135215    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 15:14:53.138342 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.138321    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 15:14:53.138508 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.138497    2582 scope.go:117] "RemoveContainer" containerID="ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6"
May 28 15:14:53.144426 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.144408    2582 scope.go:117] "RemoveContainer" containerID="5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39"
May 28 15:14:53.150904 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.150887    2582 scope.go:117] "RemoveContainer" containerID="c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea"
May 28 15:14:53.157247 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.157230    2582 scope.go:117] "RemoveContainer" containerID="539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08"
May 28 15:14:53.157494 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:53.157472    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08\": container with ID starting with 539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08 not found: ID does not exist" containerID="539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08"
May 28 15:14:53.157548 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.157503    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08"} err="failed to get container status \"539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08\": rpc error: code = NotFound desc = could not find container \"539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08\": container with ID starting with 539d1be0acbbaa690a8fe14f3ead68943d78f4cf459fa999d73d200f5d071e08 not found: ID does not exist"
May 28 15:14:53.157548 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.157520    2582 scope.go:117] "RemoveContainer" containerID="935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c"
May 28 15:14:53.157710 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:53.157697    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c\": container with ID starting with 935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c not found: ID does not exist" containerID="935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c"
May 28 15:14:53.157748 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.157712    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c"} err="failed to get container status \"935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c\": rpc error: code = NotFound desc = could not find container \"935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c\": container with ID starting with 935d7b47a0c44b666b1ee1864183b25e2d07067ffba5ca825dceae0cce4e441c not found: ID does not exist"
May 28 15:14:53.157748 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.157724    2582 scope.go:117] "RemoveContainer" containerID="4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f"
May 28 15:14:53.157972 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:53.157953    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f\": container with ID starting with 4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f not found: ID does not exist" containerID="4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f"
May 28 15:14:53.158014 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.157980    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f"} err="failed to get container status \"4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f\": rpc error: code = NotFound desc = could not find container \"4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f\": container with ID starting with 4805d8bd8e58bed5ce1ca898385ea18ad3c1cec121100763611875357346ba3f not found: ID does not exist"
May 28 15:14:53.158014 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.157999    2582 scope.go:117] "RemoveContainer" containerID="c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4"
May 28 15:14:53.158269 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:53.158252    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4\": container with ID starting with c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4 not found: ID does not exist" containerID="c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4"
May 28 15:14:53.158311 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.158274    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4"} err="failed to get container status \"c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4\": rpc error: code = NotFound desc = could not find container \"c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4\": container with ID starting with c0cf995c7ca742f501a1400ca10e49543ff72e8a91185ad1b542b3a3c24028d4 not found: ID does not exist"
May 28 15:14:53.158311 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.158289    2582 scope.go:117] "RemoveContainer" containerID="ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6"
May 28 15:14:53.158510 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:53.158493    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6\": container with ID starting with ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6 not found: ID does not exist" containerID="ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6"
May 28 15:14:53.158550 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.158514    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6"} err="failed to get container status \"ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6\": rpc error: code = NotFound desc = could not find container \"ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6\": container with ID starting with ebbc6c3281bf7a05e45c6832e1879bc1cb3aa0d972be167cabb70ca1edd046b6 not found: ID does not exist"
May 28 15:14:53.158550 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.158528    2582 scope.go:117] "RemoveContainer" containerID="5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39"
May 28 15:14:53.158739 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:53.158722    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39\": container with ID starting with 5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39 not found: ID does not exist" containerID="5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39"
May 28 15:14:53.158784 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.158743    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39"} err="failed to get container status \"5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39\": rpc error: code = NotFound desc = could not find container \"5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39\": container with ID starting with 5a3420776536ac6a059e48768d32f0eb9579eb0442069bf77530006c5b7c2f39 not found: ID does not exist"
May 28 15:14:53.158784 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.158758    2582 scope.go:117] "RemoveContainer" containerID="c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea"
May 28 15:14:53.158941 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:14:53.158928    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea\": container with ID starting with c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea not found: ID does not exist" containerID="c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea"
May 28 15:14:53.158977 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.158942    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea"} err="failed to get container status \"c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea\": rpc error: code = NotFound desc = could not find container \"c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea\": container with ID starting with c7c16780776b668009c6b48ccad1869f6d953c95c76e666ac03db28bff9dd4ea not found: ID does not exist"
May 28 15:14:53.161802 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.161783    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 15:14:53.162053 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162041    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="prometheus"
May 28 15:14:53.162090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162055    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="prometheus"
May 28 15:14:53.162090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162065    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy-web"
May 28 15:14:53.162090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162073    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy-web"
May 28 15:14:53.162090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162084    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy"
May 28 15:14:53.162090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162089    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy"
May 28 15:14:53.162272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162106    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="config-reloader"
May 28 15:14:53.162272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162112    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="config-reloader"
May 28 15:14:53.162272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162199    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="thanos-sidecar"
May 28 15:14:53.162272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162206    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="thanos-sidecar"
May 28 15:14:53.162272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162213    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy-thanos"
May 28 15:14:53.162272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162218    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy-thanos"
May 28 15:14:53.162272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162229    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="init-config-reloader"
May 28 15:14:53.162272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162234    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="init-config-reloader"
May 28 15:14:53.162501 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162285    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="config-reloader"
May 28 15:14:53.162501 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162293    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy-thanos"
May 28 15:14:53.162501 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162300    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy"
May 28 15:14:53.162501 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162307    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="thanos-sidecar"
May 28 15:14:53.162501 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162314    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="kube-rbac-proxy-web"
May 28 15:14:53.162501 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.162320    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" containerName="prometheus"
May 28 15:14:53.168587 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.168565    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.171146 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171109    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 28 15:14:53.171256 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171147    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 28 15:14:53.171453 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171431    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 28 15:14:53.171543 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171499    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a8dcglaejdj8r\""
May 28 15:14:53.171610 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171544    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 28 15:14:53.171610 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171562    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 28 15:14:53.171700 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171629    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 28 15:14:53.171700 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171653    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 28 15:14:53.171876 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171860    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 28 15:14:53.171936 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171877    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 28 15:14:53.171936 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.171867    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d7nbj\""
May 28 15:14:53.172482 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.172464    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 28 15:14:53.179809 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.179783    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 28 15:14:53.180850 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.180828    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 28 15:14:53.181269 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.181248    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 15:14:53.234323 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234323 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234537 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234537 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-web-config\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234537 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234537 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234537 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234537 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234537 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234537 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a26bc17d-d3e4-483e-b945-9384c9b63980-config-out\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234794 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a26bc17d-d3e4-483e-b945-9384c9b63980-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234794 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a26bc17d-d3e4-483e-b945-9384c9b63980-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234794 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-config\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234794 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234794 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234696    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrxt8\" (UniqueName: \"kubernetes.io/projected/a26bc17d-d3e4-483e-b945-9384c9b63980-kube-api-access-nrxt8\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234794 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234794 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.234794 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.234769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.335832 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.335793    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nrxt8\" (UniqueName: \"kubernetes.io/projected/a26bc17d-d3e4-483e-b945-9384c9b63980-kube-api-access-nrxt8\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336020 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.335863    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336020 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.335891    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336020 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.335915    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336020 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.335961    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336020 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.335984    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336020 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336020    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336321 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336221    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-web-config\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336321 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336259    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336321 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336294    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336476 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336325    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336476 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336349    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336476 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336370    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336476 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336399    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a26bc17d-d3e4-483e-b945-9384c9b63980-config-out\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336476 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336443    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a26bc17d-d3e4-483e-b945-9384c9b63980-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336476 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336470    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a26bc17d-d3e4-483e-b945-9384c9b63980-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336754 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336500    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-config\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336754 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336572    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336852 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336751    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.336852 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.336757    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.337102 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.337076    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.339165 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.339142    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.339305 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.339275    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.339305 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.339298    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.339460 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.339332    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.339535 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.339513    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a26bc17d-d3e4-483e-b945-9384c9b63980-config-out\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.339703 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.339681    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.339988 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.339963    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a26bc17d-d3e4-483e-b945-9384c9b63980-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.340159 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.340109    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.340395 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.340370    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a26bc17d-d3e4-483e-b945-9384c9b63980-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.340473 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.340376    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a26bc17d-d3e4-483e-b945-9384c9b63980-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.341481 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.341456    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.341553 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.341480    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-config\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.341735 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.341716    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-web-config\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.341886 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.341869    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a26bc17d-d3e4-483e-b945-9384c9b63980-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.343245 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.343227    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrxt8\" (UniqueName: \"kubernetes.io/projected/a26bc17d-d3e4-483e-b945-9384c9b63980-kube-api-access-nrxt8\") pod \"prometheus-k8s-0\" (UID: \"a26bc17d-d3e4-483e-b945-9384c9b63980\") " pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.480774 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.480469    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:53.609391 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:53.609365    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 28 15:14:53.611230 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:14:53.611193    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26bc17d_d3e4_483e_b945_9384c9b63980.slice/crio-702ccb15bcd1004f6e18187f49a0fbccef6e1a1ed24d14fbfb94eb2060632dc7 WatchSource:0}: Error finding container 702ccb15bcd1004f6e18187f49a0fbccef6e1a1ed24d14fbfb94eb2060632dc7: Status 404 returned error can't find the container with id 702ccb15bcd1004f6e18187f49a0fbccef6e1a1ed24d14fbfb94eb2060632dc7
May 28 15:14:54.116620 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:54.116590    2582 generic.go:358] "Generic (PLEG): container finished" podID="a26bc17d-d3e4-483e-b945-9384c9b63980" containerID="abd3d8ede31567d06fa31f058f602ff531191836af07a39efee35867e9df0912" exitCode=0
May 28 15:14:54.116994 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:54.116641    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a26bc17d-d3e4-483e-b945-9384c9b63980","Type":"ContainerDied","Data":"abd3d8ede31567d06fa31f058f602ff531191836af07a39efee35867e9df0912"}
May 28 15:14:54.116994 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:54.116660    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a26bc17d-d3e4-483e-b945-9384c9b63980","Type":"ContainerStarted","Data":"702ccb15bcd1004f6e18187f49a0fbccef6e1a1ed24d14fbfb94eb2060632dc7"}
May 28 15:14:54.293294 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:54.293268    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d04e72f0-ca2e-4ebc-b795-f7a98afa3a22" path="/var/lib/kubelet/pods/d04e72f0-ca2e-4ebc-b795-f7a98afa3a22/volumes"
May 28 15:14:55.122185 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:55.122152    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a26bc17d-d3e4-483e-b945-9384c9b63980","Type":"ContainerStarted","Data":"43c13ca381ce8cc10643c43adb96e6358b8298289b5d6b65cd1a884964209f2e"}
May 28 15:14:55.122185 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:55.122190    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a26bc17d-d3e4-483e-b945-9384c9b63980","Type":"ContainerStarted","Data":"7998aca0b5f530eaacee7d794c978c287a75eb943d0a4a86594ca52abdc64c99"}
May 28 15:14:55.122634 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:55.122203    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a26bc17d-d3e4-483e-b945-9384c9b63980","Type":"ContainerStarted","Data":"927b89a39b82d5b11422c8f6b88a2774bd5c1e325e56f512719fb7679dfc5921"}
May 28 15:14:55.122634 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:55.122215    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a26bc17d-d3e4-483e-b945-9384c9b63980","Type":"ContainerStarted","Data":"1ab462b8a4410b6e6746fb0bb38c6a1fdc92a242b9ed7bd73daf4bcca9826ef3"}
May 28 15:14:55.122634 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:55.122226    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a26bc17d-d3e4-483e-b945-9384c9b63980","Type":"ContainerStarted","Data":"a57e2740436a3ee1b0fea061397e3520ae5a25d8d3329e6f69b937baf9547048"}
May 28 15:14:55.122634 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:55.122235    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a26bc17d-d3e4-483e-b945-9384c9b63980","Type":"ContainerStarted","Data":"91c49a6ace33372bea935733949311d7d003068d4ed07bca4a9614104cf0c6b4"}
May 28 15:14:55.149720 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:55.149670    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.149655556 podStartE2EDuration="2.149655556s" podCreationTimestamp="2026-05-28 15:14:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:14:55.147234184 +0000 UTC m=+247.446303692" watchObservedRunningTime="2026-05-28 15:14:55.149655556 +0000 UTC m=+247.448725071"
May 28 15:14:58.481259 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:58.481156    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:14:59.182951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:59.182903    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:14:59.185244 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:59.185225    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60775ca8-b4ea-4176-bda3-58ad85bbc806-metrics-certs\") pod \"network-metrics-daemon-9lnwm\" (UID: \"60775ca8-b4ea-4176-bda3-58ad85bbc806\") " pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:14:59.392096 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:59.392063    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-49m45\""
May 28 15:14:59.400204 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:59.400183    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lnwm"
May 28 15:14:59.522788 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:14:59.522754    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9lnwm"]
May 28 15:14:59.525684 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:14:59.525657    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60775ca8_b4ea_4176_bda3_58ad85bbc806.slice/crio-2e589db851ab902b34071bc81bd6549fbeb50c0e3a82f9cd50ca1c1c6424eeb8 WatchSource:0}: Error finding container 2e589db851ab902b34071bc81bd6549fbeb50c0e3a82f9cd50ca1c1c6424eeb8: Status 404 returned error can't find the container with id 2e589db851ab902b34071bc81bd6549fbeb50c0e3a82f9cd50ca1c1c6424eeb8
May 28 15:15:00.139692 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:00.139654    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9lnwm" event={"ID":"60775ca8-b4ea-4176-bda3-58ad85bbc806","Type":"ContainerStarted","Data":"2e589db851ab902b34071bc81bd6549fbeb50c0e3a82f9cd50ca1c1c6424eeb8"}
May 28 15:15:01.144136 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:01.144092    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9lnwm" event={"ID":"60775ca8-b4ea-4176-bda3-58ad85bbc806","Type":"ContainerStarted","Data":"72e71b622af36578676d12cd415d54c811a0d744c1ceb9a87492a022e6dbe48e"}
May 28 15:15:01.144482 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:01.144153    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9lnwm" event={"ID":"60775ca8-b4ea-4176-bda3-58ad85bbc806","Type":"ContainerStarted","Data":"109f0ec1ca4cec86304be7916e235d46c9d9ea6a0d528228b03c3a6c1eb7373b"}
May 28 15:15:12.451384 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.451328    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9lnwm" podStartSLOduration=263.460877617 podStartE2EDuration="4m24.451313739s" podCreationTimestamp="2026-05-28 15:10:48 +0000 UTC" firstStartedPulling="2026-05-28 15:14:59.527454518 +0000 UTC m=+251.826524007" lastFinishedPulling="2026-05-28 15:15:00.51789064 +0000 UTC m=+252.816960129" observedRunningTime="2026-05-28 15:15:01.160136692 +0000 UTC m=+253.459206213" watchObservedRunningTime="2026-05-28 15:15:12.451313739 +0000 UTC m=+264.750383253"
May 28 15:15:12.452236 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.452220    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jlddv"]
May 28 15:15:12.458369 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.458344    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.460857 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.460824    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 28 15:15:12.461897 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.461874    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jlddv"]
May 28 15:15:12.609723 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.609687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61cd75b2-d230-4aab-bfe8-8b8727742281-original-pull-secret\") pod \"global-pull-secret-syncer-jlddv\" (UID: \"61cd75b2-d230-4aab-bfe8-8b8727742281\") " pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.609897 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.609746    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61cd75b2-d230-4aab-bfe8-8b8727742281-dbus\") pod \"global-pull-secret-syncer-jlddv\" (UID: \"61cd75b2-d230-4aab-bfe8-8b8727742281\") " pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.609897 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.609834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61cd75b2-d230-4aab-bfe8-8b8727742281-kubelet-config\") pod \"global-pull-secret-syncer-jlddv\" (UID: \"61cd75b2-d230-4aab-bfe8-8b8727742281\") " pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.711303 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.711212    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61cd75b2-d230-4aab-bfe8-8b8727742281-original-pull-secret\") pod \"global-pull-secret-syncer-jlddv\" (UID: \"61cd75b2-d230-4aab-bfe8-8b8727742281\") " pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.711303 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.711274    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61cd75b2-d230-4aab-bfe8-8b8727742281-dbus\") pod \"global-pull-secret-syncer-jlddv\" (UID: \"61cd75b2-d230-4aab-bfe8-8b8727742281\") " pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.711497 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.711309    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61cd75b2-d230-4aab-bfe8-8b8727742281-kubelet-config\") pod \"global-pull-secret-syncer-jlddv\" (UID: \"61cd75b2-d230-4aab-bfe8-8b8727742281\") " pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.711497 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.711384    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61cd75b2-d230-4aab-bfe8-8b8727742281-kubelet-config\") pod \"global-pull-secret-syncer-jlddv\" (UID: \"61cd75b2-d230-4aab-bfe8-8b8727742281\") " pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.711497 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.711473    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61cd75b2-d230-4aab-bfe8-8b8727742281-dbus\") pod \"global-pull-secret-syncer-jlddv\" (UID: \"61cd75b2-d230-4aab-bfe8-8b8727742281\") " pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.713386 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.713367    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61cd75b2-d230-4aab-bfe8-8b8727742281-original-pull-secret\") pod \"global-pull-secret-syncer-jlddv\" (UID: \"61cd75b2-d230-4aab-bfe8-8b8727742281\") " pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.767745 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.767708    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlddv"
May 28 15:15:12.885928 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:12.885901    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jlddv"]
May 28 15:15:12.888858 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:15:12.888826    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cd75b2_d230_4aab_bfe8_8b8727742281.slice/crio-c5dff26050225b20963338f2197f87860149278608a8d6ed53e6767e153d084d WatchSource:0}: Error finding container c5dff26050225b20963338f2197f87860149278608a8d6ed53e6767e153d084d: Status 404 returned error can't find the container with id c5dff26050225b20963338f2197f87860149278608a8d6ed53e6767e153d084d
May 28 15:15:13.182210 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:13.182174    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jlddv" event={"ID":"61cd75b2-d230-4aab-bfe8-8b8727742281","Type":"ContainerStarted","Data":"c5dff26050225b20963338f2197f87860149278608a8d6ed53e6767e153d084d"}
May 28 15:15:17.202060 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:17.202019    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jlddv" event={"ID":"61cd75b2-d230-4aab-bfe8-8b8727742281","Type":"ContainerStarted","Data":"edde3dbf5ccd5c9d29186b158135358c399781e4682cca25e71a3725ad7296e4"}
May 28 15:15:17.217661 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:17.217611    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jlddv" podStartSLOduration=1.271487993 podStartE2EDuration="5.217597738s" podCreationTimestamp="2026-05-28 15:15:12 +0000 UTC" firstStartedPulling="2026-05-28 15:15:12.890688183 +0000 UTC m=+265.189757672" lastFinishedPulling="2026-05-28 15:15:16.836797925 +0000 UTC m=+269.135867417" observedRunningTime="2026-05-28 15:15:17.216554853 +0000 UTC m=+269.515624366" watchObservedRunningTime="2026-05-28 15:15:17.217597738 +0000 UTC m=+269.516667245"
May 28 15:15:48.213697 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:48.213662    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:15:48.214324 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:48.213902    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:15:48.223209 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:48.223193    2582 kubelet.go:1628] "Image garbage collection succeeded"
May 28 15:15:53.481569 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:53.481535    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:15:53.496866 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:53.496841    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:15:54.328563 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:15:54.328532    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 28 15:17:19.515879 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.515835    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"]
May 28 15:17:19.520398 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.520374    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.523305 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.523012    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 15:17:19.523305 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.523100    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 15:17:19.523305 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.523135    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 15:17:19.523536 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.523383    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bcqlm\""
May 28 15:17:19.523536 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.523424    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 15:17:19.523633 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.523576    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 15:17:19.523633 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.523604    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 15:17:19.523790 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.523766    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 15:17:19.539164 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.535402    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"]
May 28 15:17:19.576900 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.576858    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-registry-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.577102 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.576919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-ui-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.577102 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.577045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-offline-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.577102 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.577086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-online-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.577276 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.577151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vdv\" (UniqueName: \"kubernetes.io/projected/90f68194-fcb9-48d8-b2b6-a03461b60e60-kube-api-access-86vdv\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.577276 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.577191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/90f68194-fcb9-48d8-b2b6-a03461b60e60-feast-data\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.678476 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.678430    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-registry-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.678674 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.678487    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-ui-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.678674 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.678546    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-offline-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.678674 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.678570    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-online-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.678674 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.678597    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-86vdv\" (UniqueName: \"kubernetes.io/projected/90f68194-fcb9-48d8-b2b6-a03461b60e60-kube-api-access-86vdv\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.678674 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.678654    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/90f68194-fcb9-48d8-b2b6-a03461b60e60-feast-data\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.678917 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:17:19.678691    2582 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 15:17:19.678917 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:17:19.678783    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-ui-tls podName:90f68194-fcb9-48d8-b2b6-a03461b60e60 nodeName:}" failed. No retries permitted until 2026-05-28 15:17:20.178760579 +0000 UTC m=+392.477830080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-ui-tls") pod "feast-simple-feast-setup-5d4d6c99fb-cvggt" (UID: "90f68194-fcb9-48d8-b2b6-a03461b60e60") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 15:17:19.679023 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.678983    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/90f68194-fcb9-48d8-b2b6-a03461b60e60-feast-data\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.679073 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:17:19.679035    2582 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 15:17:19.679153 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:17:19.679083    2582 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 28 15:17:19.679153 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:17:19.679104    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-online-tls podName:90f68194-fcb9-48d8-b2b6-a03461b60e60 nodeName:}" failed. No retries permitted until 2026-05-28 15:17:20.179086627 +0000 UTC m=+392.478156128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-online-tls") pod "feast-simple-feast-setup-5d4d6c99fb-cvggt" (UID: "90f68194-fcb9-48d8-b2b6-a03461b60e60") : secret "feast-simple-feast-setup-online-tls" not found
May 28 15:17:19.679153 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:17:19.679147    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-offline-tls podName:90f68194-fcb9-48d8-b2b6-a03461b60e60 nodeName:}" failed. No retries permitted until 2026-05-28 15:17:20.179113361 +0000 UTC m=+392.478182858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-offline-tls") pod "feast-simple-feast-setup-5d4d6c99fb-cvggt" (UID: "90f68194-fcb9-48d8-b2b6-a03461b60e60") : secret "feast-simple-feast-setup-offline-tls" not found
May 28 15:17:19.681513 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.681485    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-registry-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:19.687938 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:19.687902    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vdv\" (UniqueName: \"kubernetes.io/projected/90f68194-fcb9-48d8-b2b6-a03461b60e60-kube-api-access-86vdv\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:20.183375 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:20.183341    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-ui-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:20.183583 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:20.183398    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-offline-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:20.183658 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:20.183581    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-online-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:20.185811 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:20.185779    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-ui-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:20.185928 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:20.185830    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-online-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:20.185928 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:20.185910    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-offline-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-cvggt\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:20.443073 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:20.442973    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:17:20.577669 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:20.577638    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"]
May 28 15:17:20.579017 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:17:20.578990    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90f68194_fcb9_48d8_b2b6_a03461b60e60.slice/crio-93d89483cb44614c0f4adea0564bd65dea4ce90b70d152a5c9d70fd81aa61494 WatchSource:0}: Error finding container 93d89483cb44614c0f4adea0564bd65dea4ce90b70d152a5c9d70fd81aa61494: Status 404 returned error can't find the container with id 93d89483cb44614c0f4adea0564bd65dea4ce90b70d152a5c9d70fd81aa61494
May 28 15:17:20.580579 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:20.580562    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 15:17:21.556950 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:21.556892    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerStarted","Data":"93d89483cb44614c0f4adea0564bd65dea4ce90b70d152a5c9d70fd81aa61494"}
May 28 15:17:50.650361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:50.650323    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerStarted","Data":"4ec53ffce43d0cbcfa6add6de3ba3a75c55b6d6e539dc54926dafc91e0507ed3"}
May 28 15:17:55.667434 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:55.667400    2582 generic.go:358] "Generic (PLEG): container finished" podID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerID="4ec53ffce43d0cbcfa6add6de3ba3a75c55b6d6e539dc54926dafc91e0507ed3" exitCode=0
May 28 15:17:55.667795 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:55.667475    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerDied","Data":"4ec53ffce43d0cbcfa6add6de3ba3a75c55b6d6e539dc54926dafc91e0507ed3"}
May 28 15:17:56.672803 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:17:56.672769    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerStarted","Data":"f4426d260168088da3f563b0c0dc91fe2df93dba2f3f200d3cd0cf23cdd5a21f"}
May 28 15:18:01.689362 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:01.689329    2582 generic.go:358] "Generic (PLEG): container finished" podID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerID="f4426d260168088da3f563b0c0dc91fe2df93dba2f3f200d3cd0cf23cdd5a21f" exitCode=0
May 28 15:18:01.689721 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:01.689381    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerDied","Data":"f4426d260168088da3f563b0c0dc91fe2df93dba2f3f200d3cd0cf23cdd5a21f"}
May 28 15:18:01.699406 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:01.699383    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 15:18:02.696951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:02.696912    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerStarted","Data":"c3ad3b3a58e7f2546e4d90f9cb53151694547d3991b59c121075ed49465ccae3"}
May 28 15:18:02.696951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:02.696957    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerStarted","Data":"dd421e2d7ec64cf00f786ee7c72fbb4e3993d396f1dfd767e8902088d5c371a8"}
May 28 15:18:02.697496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:02.696972    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerStarted","Data":"20f12933e36cf30b00acdb458a2b597a84df7cf500f7fda5e8d1288ff6838fb0"}
May 28 15:18:02.697496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:02.696985    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerStarted","Data":"2104f069007591e59f4e32493736c81c279f084ae5abbb1c65b5229add6b88d4"}
May 28 15:18:02.722842 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:02.722782    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podStartSLOduration=13.774361331 podStartE2EDuration="43.722762873s" podCreationTimestamp="2026-05-28 15:17:19 +0000 UTC" firstStartedPulling="2026-05-28 15:17:20.58071367 +0000 UTC m=+392.879783159" lastFinishedPulling="2026-05-28 15:17:50.529115212 +0000 UTC m=+422.828184701" observedRunningTime="2026-05-28 15:18:02.721309164 +0000 UTC m=+435.020378675" watchObservedRunningTime="2026-05-28 15:18:02.722762873 +0000 UTC m=+435.021832383"
May 28 15:18:05.443872 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:05.443836    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:05.444420 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:05.444387    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:05.444420 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:05.444415    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:05.444420 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:05.444427    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:05.445921 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:05.445889    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 28 15:18:05.446156 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:05.445918    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused"
May 28 15:18:05.446264 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:05.445937    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused"
May 28 15:18:05.446264 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:05.445977    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 28 15:18:08.445071 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.445032    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.445479 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.445206    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.445479 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.445359    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.445809 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.445771    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.445809 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.445800    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.445809 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.445813    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.446028 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.445901    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.446028 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.445944    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.446234 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.446221    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.449627 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.449610    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.721471 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.721375    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:18:08.724298 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:18:08.724275    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:19:10.940714 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:10.940679    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"]
May 28 15:19:10.941424 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:10.941017    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="registry" containerID="cri-o://2104f069007591e59f4e32493736c81c279f084ae5abbb1c65b5229add6b88d4" gracePeriod=30
May 28 15:19:10.941424 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:10.941178    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="offline" containerID="cri-o://dd421e2d7ec64cf00f786ee7c72fbb4e3993d396f1dfd767e8902088d5c371a8" gracePeriod=30
May 28 15:19:10.941424 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:10.941189    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="online" containerID="cri-o://20f12933e36cf30b00acdb458a2b597a84df7cf500f7fda5e8d1288ff6838fb0" gracePeriod=30
May 28 15:19:10.941424 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:10.941256    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="ui" containerID="cri-o://c3ad3b3a58e7f2546e4d90f9cb53151694547d3991b59c121075ed49465ccae3" gracePeriod=30
May 28 15:19:11.303257 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.303153    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"]
May 28 15:19:11.308819 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.308789    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.311613 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.311410    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-8gnrj\""
May 28 15:19:11.318375 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.318348    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"]
May 28 15:19:11.360719 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.360683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-offline-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.360719 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.360731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea4a236-5624-40ba-895f-7fdba4a7f70f-feast-data\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.360966 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.360797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-online-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.360966 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.360871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ntdk\" (UniqueName: \"kubernetes.io/projected/9ea4a236-5624-40ba-895f-7fdba4a7f70f-kube-api-access-6ntdk\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.360966 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.360929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-registry-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.360966 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.360952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-ui-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.462194 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.462153    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-offline-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.462354 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.462203    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea4a236-5624-40ba-895f-7fdba4a7f70f-feast-data\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.462354 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.462228    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-online-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.462354 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.462263    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6ntdk\" (UniqueName: \"kubernetes.io/projected/9ea4a236-5624-40ba-895f-7fdba4a7f70f-kube-api-access-6ntdk\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.462354 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.462314    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-registry-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.462354 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.462338    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-ui-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.462618 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:19:11.462478    2582 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 15:19:11.462618 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:19:11.462537    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-ui-tls podName:9ea4a236-5624-40ba-895f-7fdba4a7f70f nodeName:}" failed. No retries permitted until 2026-05-28 15:19:11.962516062 +0000 UTC m=+504.261585552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-ui-tls") pod "feast-simple-feast-setup-5d4d6c99fb-dh5mq" (UID: "9ea4a236-5624-40ba-895f-7fdba4a7f70f") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 15:19:11.462618 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.462578    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea4a236-5624-40ba-895f-7fdba4a7f70f-feast-data\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.465082 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.465056    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-registry-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.465372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.465349    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-offline-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.465658 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.465639    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-online-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.475305 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.475279    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ntdk\" (UniqueName: \"kubernetes.io/projected/9ea4a236-5624-40ba-895f-7fdba4a7f70f-kube-api-access-6ntdk\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.911832 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.911799    2582 generic.go:358] "Generic (PLEG): container finished" podID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerID="c3ad3b3a58e7f2546e4d90f9cb53151694547d3991b59c121075ed49465ccae3" exitCode=0
May 28 15:19:11.911832 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.911828    2582 generic.go:358] "Generic (PLEG): container finished" podID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerID="20f12933e36cf30b00acdb458a2b597a84df7cf500f7fda5e8d1288ff6838fb0" exitCode=0
May 28 15:19:11.912029 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.911874    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerDied","Data":"c3ad3b3a58e7f2546e4d90f9cb53151694547d3991b59c121075ed49465ccae3"}
May 28 15:19:11.912029 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.911908    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerDied","Data":"20f12933e36cf30b00acdb458a2b597a84df7cf500f7fda5e8d1288ff6838fb0"}
May 28 15:19:11.967498 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.967467    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-ui-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:11.969727 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:11.969710    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-ui-tls\") pod \"feast-simple-feast-setup-5d4d6c99fb-dh5mq\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") " pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:12.223371 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:12.223272    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:12.354035 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:12.354010    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"]
May 28 15:19:12.355323 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:19:12.355291    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ea4a236_5624_40ba_895f_7fdba4a7f70f.slice/crio-14d47ac40efebefa9ebfc995723278094fa44050ab31ca9b4f323d0650a8f1fc WatchSource:0}: Error finding container 14d47ac40efebefa9ebfc995723278094fa44050ab31ca9b4f323d0650a8f1fc: Status 404 returned error can't find the container with id 14d47ac40efebefa9ebfc995723278094fa44050ab31ca9b4f323d0650a8f1fc
May 28 15:19:12.916713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:12.916667    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerStarted","Data":"a8fcad80e31d6532fbe40910194afbbe71125fc6c29fbeac796c73082aed92a2"}
May 28 15:19:12.916713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:12.916707    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerStarted","Data":"14d47ac40efebefa9ebfc995723278094fa44050ab31ca9b4f323d0650a8f1fc"}
May 28 15:19:16.931770 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:16.931734    2582 generic.go:358] "Generic (PLEG): container finished" podID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerID="a8fcad80e31d6532fbe40910194afbbe71125fc6c29fbeac796c73082aed92a2" exitCode=0
May 28 15:19:16.932099 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:16.931812    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerDied","Data":"a8fcad80e31d6532fbe40910194afbbe71125fc6c29fbeac796c73082aed92a2"}
May 28 15:19:17.936340 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:17.936310    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerStarted","Data":"de0b4c2a87f17e78d34baec46017b429d7b43e71e853aa0be4e22ffa2da36209"}
May 28 15:19:18.445957 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:18.445908    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 28 15:19:18.722541 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:18.722450    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 28 15:19:22.951235 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:22.951196    2582 generic.go:358] "Generic (PLEG): container finished" podID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerID="de0b4c2a87f17e78d34baec46017b429d7b43e71e853aa0be4e22ffa2da36209" exitCode=0
May 28 15:19:22.951235 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:22.951240    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerDied","Data":"de0b4c2a87f17e78d34baec46017b429d7b43e71e853aa0be4e22ffa2da36209"}
May 28 15:19:23.959579 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:23.959535    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerStarted","Data":"f34a1c8634262332c9505b82e475d6103574ce3327dde92055586c120192fae7"}
May 28 15:19:23.959579 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:23.959586    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerStarted","Data":"fda98f76cf080141dca7b9d95c606634e4027b59272e2180289f7c124122b136"}
May 28 15:19:23.960097 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:23.959599    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerStarted","Data":"737d1420b3e1da86d8484fd6cc9ff12ee7739c535b551a9735157bcbee1aa55d"}
May 28 15:19:23.960097 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:23.959611    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerStarted","Data":"3092df12e1f1477c653b75c7b813fe7fad007ee5075850c8583761255d30f202"}
May 28 15:19:23.986111 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:23.985878    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podStartSLOduration=12.985859361 podStartE2EDuration="12.985859361s" podCreationTimestamp="2026-05-28 15:19:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:19:23.984538102 +0000 UTC m=+516.283607635" watchObservedRunningTime="2026-05-28 15:19:23.985859361 +0000 UTC m=+516.284928890"
May 28 15:19:24.223614 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:24.223496    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:24.224082 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:24.224057    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:24.224220 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:24.224093    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:24.224220 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:24.224106    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:24.225594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:24.225553    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused"
May 28 15:19:24.225710 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:24.225594    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 28 15:19:24.225710 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:24.225607    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 28 15:19:24.225710 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:24.225674    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused"
May 28 15:19:27.223986 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:27.223885    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 28 15:19:27.224527 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:27.223892    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused"
May 28 15:19:27.224716 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:27.223884    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused"
May 28 15:19:27.224831 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:27.223930    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 28 15:19:28.446288 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:28.446241    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 28 15:19:28.722093 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:28.721992    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 28 15:19:30.225439 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.225406    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.225795 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.225535    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.225795 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.225744    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.226057 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.226031    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.226205 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.226063    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.226205 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.226078    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.226420 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.226399    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.226536 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.226502    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.226597 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.226572    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.229862 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.229843    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.984100 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.984071    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:30.986898 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:30.986875    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:19:38.446521 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:38.446474    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 28 15:19:38.446949 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:38.446611    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:19:38.722175 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:38.722065    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 28 15:19:38.722327 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:38.722211    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:19:41.022840 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.022803    2582 generic.go:358] "Generic (PLEG): container finished" podID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerID="dd421e2d7ec64cf00f786ee7c72fbb4e3993d396f1dfd767e8902088d5c371a8" exitCode=137
May 28 15:19:41.022840 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.022832    2582 generic.go:358] "Generic (PLEG): container finished" podID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerID="2104f069007591e59f4e32493736c81c279f084ae5abbb1c65b5229add6b88d4" exitCode=137
May 28 15:19:41.023213 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.022856    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerDied","Data":"dd421e2d7ec64cf00f786ee7c72fbb4e3993d396f1dfd767e8902088d5c371a8"}
May 28 15:19:41.023213 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.022877    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerDied","Data":"2104f069007591e59f4e32493736c81c279f084ae5abbb1c65b5229add6b88d4"}
May 28 15:19:41.585478 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.585453    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:19:41.732168 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.732072    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/90f68194-fcb9-48d8-b2b6-a03461b60e60-feast-data\") pod \"90f68194-fcb9-48d8-b2b6-a03461b60e60\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") "
May 28 15:19:41.732168 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.732116    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-online-tls\") pod \"90f68194-fcb9-48d8-b2b6-a03461b60e60\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") "
May 28 15:19:41.732466 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.732170    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86vdv\" (UniqueName: \"kubernetes.io/projected/90f68194-fcb9-48d8-b2b6-a03461b60e60-kube-api-access-86vdv\") pod \"90f68194-fcb9-48d8-b2b6-a03461b60e60\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") "
May 28 15:19:41.732466 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.732201    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-offline-tls\") pod \"90f68194-fcb9-48d8-b2b6-a03461b60e60\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") "
May 28 15:19:41.732466 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.732223    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-ui-tls\") pod \"90f68194-fcb9-48d8-b2b6-a03461b60e60\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") "
May 28 15:19:41.732466 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.732253    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-registry-tls\") pod \"90f68194-fcb9-48d8-b2b6-a03461b60e60\" (UID: \"90f68194-fcb9-48d8-b2b6-a03461b60e60\") "
May 28 15:19:41.732631 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.732611    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f68194-fcb9-48d8-b2b6-a03461b60e60-feast-data" (OuterVolumeSpecName: "feast-data") pod "90f68194-fcb9-48d8-b2b6-a03461b60e60" (UID: "90f68194-fcb9-48d8-b2b6-a03461b60e60"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:19:41.734452 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.734422    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "90f68194-fcb9-48d8-b2b6-a03461b60e60" (UID: "90f68194-fcb9-48d8-b2b6-a03461b60e60"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:19:41.734766 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.734736    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "90f68194-fcb9-48d8-b2b6-a03461b60e60" (UID: "90f68194-fcb9-48d8-b2b6-a03461b60e60"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:19:41.734873 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.734787    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-online-tls" (OuterVolumeSpecName: "online-tls") pod "90f68194-fcb9-48d8-b2b6-a03461b60e60" (UID: "90f68194-fcb9-48d8-b2b6-a03461b60e60"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:19:41.734873 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.734787    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f68194-fcb9-48d8-b2b6-a03461b60e60-kube-api-access-86vdv" (OuterVolumeSpecName: "kube-api-access-86vdv") pod "90f68194-fcb9-48d8-b2b6-a03461b60e60" (UID: "90f68194-fcb9-48d8-b2b6-a03461b60e60"). InnerVolumeSpecName "kube-api-access-86vdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:19:41.734873 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.734804    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "90f68194-fcb9-48d8-b2b6-a03461b60e60" (UID: "90f68194-fcb9-48d8-b2b6-a03461b60e60"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:19:41.832865 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.832836    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-registry-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:19:41.832865 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.832860    2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/90f68194-fcb9-48d8-b2b6-a03461b60e60-feast-data\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:19:41.832865 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.832868    2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-online-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:19:41.833065 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.832876    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86vdv\" (UniqueName: \"kubernetes.io/projected/90f68194-fcb9-48d8-b2b6-a03461b60e60-kube-api-access-86vdv\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:19:41.833065 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.832885    2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-offline-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:19:41.833065 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:41.832894    2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/90f68194-fcb9-48d8-b2b6-a03461b60e60-ui-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:19:42.028231 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.028163    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"
May 28 15:19:42.028580 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.028159    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt" event={"ID":"90f68194-fcb9-48d8-b2b6-a03461b60e60","Type":"ContainerDied","Data":"93d89483cb44614c0f4adea0564bd65dea4ce90b70d152a5c9d70fd81aa61494"}
May 28 15:19:42.028580 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.028277    2582 scope.go:117] "RemoveContainer" containerID="c3ad3b3a58e7f2546e4d90f9cb53151694547d3991b59c121075ed49465ccae3"
May 28 15:19:42.036800 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.036778    2582 scope.go:117] "RemoveContainer" containerID="dd421e2d7ec64cf00f786ee7c72fbb4e3993d396f1dfd767e8902088d5c371a8"
May 28 15:19:42.043820 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.043806    2582 scope.go:117] "RemoveContainer" containerID="20f12933e36cf30b00acdb458a2b597a84df7cf500f7fda5e8d1288ff6838fb0"
May 28 15:19:42.050590 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.050576    2582 scope.go:117] "RemoveContainer" containerID="2104f069007591e59f4e32493736c81c279f084ae5abbb1c65b5229add6b88d4"
May 28 15:19:42.057962 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.057943    2582 scope.go:117] "RemoveContainer" containerID="f4426d260168088da3f563b0c0dc91fe2df93dba2f3f200d3cd0cf23cdd5a21f"
May 28 15:19:42.061420 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.061404    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"]
May 28 15:19:42.062659 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.062639    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-cvggt"]
May 28 15:19:42.064995 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.064979    2582 scope.go:117] "RemoveContainer" containerID="4ec53ffce43d0cbcfa6add6de3ba3a75c55b6d6e539dc54926dafc91e0507ed3"
May 28 15:19:42.292368 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:19:42.292291    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" path="/var/lib/kubelet/pods/90f68194-fcb9-48d8-b2b6-a03461b60e60/volumes"
May 28 15:20:32.953097 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953062    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"]
May 28 15:20:32.953604 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953538    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="registry"
May 28 15:20:32.953604 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953554    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="registry"
May 28 15:20:32.953604 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953565    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="online"
May 28 15:20:32.953604 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953573    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="online"
May 28 15:20:32.953604 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953583    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="feast-init"
May 28 15:20:32.953604 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953592    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="feast-init"
May 28 15:20:32.953604 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953607    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="feast-apply"
May 28 15:20:32.953854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953615    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="feast-apply"
May 28 15:20:32.953854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953625    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="offline"
May 28 15:20:32.953854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953633    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="offline"
May 28 15:20:32.953854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953664    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="ui"
May 28 15:20:32.953854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953672    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="ui"
May 28 15:20:32.953854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953738    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="online"
May 28 15:20:32.953854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953750    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="offline"
May 28 15:20:32.953854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953761    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="registry"
May 28 15:20:32.953854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.953772    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="90f68194-fcb9-48d8-b2b6-a03461b60e60" containerName="ui"
May 28 15:20:32.957479 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.957462    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:32.960361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.960038    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 15:20:32.960361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.960097    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-tnzc5\""
May 28 15:20:32.960361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.960166    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 15:20:32.960361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.960100    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 15:20:32.960361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.960222    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 15:20:32.960361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.960039    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 15:20:32.960361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.960169    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 28 15:20:32.960361 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.960345    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 15:20:32.974404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:32.974382    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"]
May 28 15:20:33.056141 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.056083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-registry-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.056364 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.056159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-offline-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.056364 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.056212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcftn\" (UniqueName: \"kubernetes.io/projected/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-kube-api-access-hcftn\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.056364 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.056248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-online-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.056364 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.056344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-feast-data\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.056513 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.056371    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-ui-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.157656 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.157620    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-feast-data\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.157656 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.157659    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-ui-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.157900 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.157698    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-registry-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.157900 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.157776    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-offline-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.157900 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.157836    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hcftn\" (UniqueName: \"kubernetes.io/projected/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-kube-api-access-hcftn\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.157900 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.157873    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-online-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.158100 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:20:33.157902    2582 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 15:20:33.158100 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:20:33.157987    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-offline-tls podName:95995db6-d3d0-4870-9b88-8f1f8b7ab77d nodeName:}" failed. No retries permitted until 2026-05-28 15:20:33.657964908 +0000 UTC m=+585.957034397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-offline-tls") pod "feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" (UID: "95995db6-d3d0-4870-9b88-8f1f8b7ab77d") : secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 15:20:33.158244 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.158101    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-feast-data\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.158483 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.158465    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-registry-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.160402 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.160381    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-ui-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.160490 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.160384    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-online-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.165799 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.165780    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcftn\" (UniqueName: \"kubernetes.io/projected/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-kube-api-access-hcftn\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.663179 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.663109    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-offline-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.665535 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.665509    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-offline-tls\") pod \"feast-simple-feast-remote-setup-b4d59d8bd-9qx2g\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:33.874678 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:33.874646    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:34.002608 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:34.002580    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"]
May 28 15:20:34.004714 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:20:34.004689    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95995db6_d3d0_4870_9b88_8f1f8b7ab77d.slice/crio-9c2c5aa94be75adfaf2c0a8cf10456caee1b534b176fea4ad40fd9d18fa3ce48 WatchSource:0}: Error finding container 9c2c5aa94be75adfaf2c0a8cf10456caee1b534b176fea4ad40fd9d18fa3ce48: Status 404 returned error can't find the container with id 9c2c5aa94be75adfaf2c0a8cf10456caee1b534b176fea4ad40fd9d18fa3ce48
May 28 15:20:34.189422 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:34.189330    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerStarted","Data":"984631ade658e189465b350f2cd596ad7ad773718586b55fa5d066530664ae36"}
May 28 15:20:34.189422 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:34.189371    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerStarted","Data":"9c2c5aa94be75adfaf2c0a8cf10456caee1b534b176fea4ad40fd9d18fa3ce48"}
May 28 15:20:39.205963 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:39.205927    2582 generic.go:358] "Generic (PLEG): container finished" podID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerID="984631ade658e189465b350f2cd596ad7ad773718586b55fa5d066530664ae36" exitCode=0
May 28 15:20:39.206359 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:39.205999    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerDied","Data":"984631ade658e189465b350f2cd596ad7ad773718586b55fa5d066530664ae36"}
May 28 15:20:40.210788 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:40.210757    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerStarted","Data":"10cf1a1d36a33d85d7ed320d13ac4dea0a9c9c4765677ed415c6a918e6b0446a"}
May 28 15:20:45.230339 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:45.230304    2582 generic.go:358] "Generic (PLEG): container finished" podID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerID="10cf1a1d36a33d85d7ed320d13ac4dea0a9c9c4765677ed415c6a918e6b0446a" exitCode=0
May 28 15:20:45.230727 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:45.230362    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerDied","Data":"10cf1a1d36a33d85d7ed320d13ac4dea0a9c9c4765677ed415c6a918e6b0446a"}
May 28 15:20:46.237201 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:46.237159    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerStarted","Data":"fd21bf53b8f0a7abb01fdcff7c7bbc47d165f09bc2f3fcd4e3cb6f0a6c897d81"}
May 28 15:20:46.237544 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:46.237216    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerStarted","Data":"af48f1c3daf214a98d1c6a1ab391cfb107de10cb62b306e91e79cb616cb098da"}
May 28 15:20:46.237544 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:46.237232    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerStarted","Data":"4b9e8c737c38b5725d5a3372ff0dc8af47afa21c7fcde03acc75f6229134b7c0"}
May 28 15:20:46.261099 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:46.261043    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podStartSLOduration=14.261026959 podStartE2EDuration="14.261026959s" podCreationTimestamp="2026-05-28 15:20:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:20:46.258313742 +0000 UTC m=+598.557383259" watchObservedRunningTime="2026-05-28 15:20:46.261026959 +0000 UTC m=+598.560096470"
May 28 15:20:48.241680 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:48.241647    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:20:48.244742 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:48.244714    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:20:48.874852 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:48.874822    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:48.875048 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:48.874943    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:48.875048 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:48.874960    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:48.876820 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:48.876777    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused"
May 28 15:20:48.876948 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:48.876831    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused"
May 28 15:20:48.876948 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:48.876876    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused"
May 28 15:20:51.875966 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:51.875937    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:51.876479 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:51.876116    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:51.876578 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:51.876521    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:51.876648 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:51.876592    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:51.876968 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:51.876938    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:51.877188 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:51.876980    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:51.880260 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:51.880242    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:52.256909 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:52.256820    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:20:52.259744 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:20:52.259715    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:21:54.271705 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:54.271666    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"]
May 28 15:21:54.272251 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:54.272073    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="online" containerID="cri-o://4b9e8c737c38b5725d5a3372ff0dc8af47afa21c7fcde03acc75f6229134b7c0" gracePeriod=30
May 28 15:21:54.272251 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:54.272152    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="ui" containerID="cri-o://fd21bf53b8f0a7abb01fdcff7c7bbc47d165f09bc2f3fcd4e3cb6f0a6c897d81" gracePeriod=30
May 28 15:21:54.272375 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:54.272166    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="offline" containerID="cri-o://af48f1c3daf214a98d1c6a1ab391cfb107de10cb62b306e91e79cb616cb098da" gracePeriod=30
May 28 15:21:54.403856 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:54.403820    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"]
May 28 15:21:54.404429 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:54.404253    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="registry" containerID="cri-o://3092df12e1f1477c653b75c7b813fe7fad007ee5075850c8583761255d30f202" gracePeriod=30
May 28 15:21:54.404429 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:54.404308    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="offline" containerID="cri-o://fda98f76cf080141dca7b9d95c606634e4027b59272e2180289f7c124122b136" gracePeriod=30
May 28 15:21:54.404429 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:54.404335    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="online" containerID="cri-o://737d1420b3e1da86d8484fd6cc9ff12ee7739c535b551a9735157bcbee1aa55d" gracePeriod=30
May 28 15:21:54.404429 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:54.404410    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="ui" containerID="cri-o://f34a1c8634262332c9505b82e475d6103574ce3327dde92055586c120192fae7" gracePeriod=30
May 28 15:21:55.446759 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:55.446724    2582 generic.go:358] "Generic (PLEG): container finished" podID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerID="f34a1c8634262332c9505b82e475d6103574ce3327dde92055586c120192fae7" exitCode=0
May 28 15:21:55.446759 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:55.446750    2582 generic.go:358] "Generic (PLEG): container finished" podID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerID="737d1420b3e1da86d8484fd6cc9ff12ee7739c535b551a9735157bcbee1aa55d" exitCode=0
May 28 15:21:55.447189 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:55.446792    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerDied","Data":"f34a1c8634262332c9505b82e475d6103574ce3327dde92055586c120192fae7"}
May 28 15:21:55.447189 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:55.446822    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerDied","Data":"737d1420b3e1da86d8484fd6cc9ff12ee7739c535b551a9735157bcbee1aa55d"}
May 28 15:21:55.449043 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:55.449023    2582 generic.go:358] "Generic (PLEG): container finished" podID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerID="fd21bf53b8f0a7abb01fdcff7c7bbc47d165f09bc2f3fcd4e3cb6f0a6c897d81" exitCode=0
May 28 15:21:55.449043 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:55.449041    2582 generic.go:358] "Generic (PLEG): container finished" podID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerID="4b9e8c737c38b5725d5a3372ff0dc8af47afa21c7fcde03acc75f6229134b7c0" exitCode=0
May 28 15:21:55.449142 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:55.449089    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerDied","Data":"fd21bf53b8f0a7abb01fdcff7c7bbc47d165f09bc2f3fcd4e3cb6f0a6c897d81"}
May 28 15:21:55.449142 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:21:55.449113    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerDied","Data":"4b9e8c737c38b5725d5a3372ff0dc8af47afa21c7fcde03acc75f6229134b7c0"}
May 28 15:22:00.226728 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:00.226678    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 28 15:22:00.984957 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:00.984914    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 28 15:22:01.876628 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:01.876586    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused"
May 28 15:22:02.257600 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:02.257495    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused"
May 28 15:22:10.226469 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:10.226421    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 28 15:22:10.985199 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:10.985157    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 28 15:22:11.877480 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:11.877440    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused"
May 28 15:22:12.257857 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:12.257762    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused"
May 28 15:22:20.226980 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:20.226936    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 28 15:22:20.227434 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:20.227077    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:22:20.985215 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:20.985171    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 28 15:22:20.985403 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:20.985304    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:22:21.877198 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:21.877156    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused"
May 28 15:22:21.877605 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:21.877278    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:22:22.257854 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:22.257770    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused"
May 28 15:22:22.258002 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:22.257876    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:22:24.535747 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:24.535666    2582 generic.go:358] "Generic (PLEG): container finished" podID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerID="fda98f76cf080141dca7b9d95c606634e4027b59272e2180289f7c124122b136" exitCode=137
May 28 15:22:24.535747 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:24.535692    2582 generic.go:358] "Generic (PLEG): container finished" podID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerID="3092df12e1f1477c653b75c7b813fe7fad007ee5075850c8583761255d30f202" exitCode=137
May 28 15:22:24.535747 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:24.535734    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerDied","Data":"fda98f76cf080141dca7b9d95c606634e4027b59272e2180289f7c124122b136"}
May 28 15:22:24.536256 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:24.535766    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerDied","Data":"3092df12e1f1477c653b75c7b813fe7fad007ee5075850c8583761255d30f202"}
May 28 15:22:24.537960 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:24.537924    2582 generic.go:358] "Generic (PLEG): container finished" podID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerID="af48f1c3daf214a98d1c6a1ab391cfb107de10cb62b306e91e79cb616cb098da" exitCode=137
May 28 15:22:24.538089 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:24.538006    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerDied","Data":"af48f1c3daf214a98d1c6a1ab391cfb107de10cb62b306e91e79cb616cb098da"}
May 28 15:22:24.924600 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:24.924575    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:22:25.021566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.021535    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-registry-tls\") pod \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") "
May 28 15:22:25.021727 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.021605    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-ui-tls\") pod \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") "
May 28 15:22:25.021727 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.021633    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-offline-tls\") pod \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") "
May 28 15:22:25.021727 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.021648    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-online-tls\") pod \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") "
May 28 15:22:25.021727 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.021704    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-feast-data\") pod \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") "
May 28 15:22:25.021944 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.021779    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcftn\" (UniqueName: \"kubernetes.io/projected/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-kube-api-access-hcftn\") pod \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\" (UID: \"95995db6-d3d0-4870-9b88-8f1f8b7ab77d\") "
May 28 15:22:25.022007 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.021950    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "95995db6-d3d0-4870-9b88-8f1f8b7ab77d" (UID: "95995db6-d3d0-4870-9b88-8f1f8b7ab77d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 15:22:25.022181 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.022160    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-registry-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.022422 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.022373    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-feast-data" (OuterVolumeSpecName: "feast-data") pod "95995db6-d3d0-4870-9b88-8f1f8b7ab77d" (UID: "95995db6-d3d0-4870-9b88-8f1f8b7ab77d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:22:25.023986 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.023955    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-online-tls" (OuterVolumeSpecName: "online-tls") pod "95995db6-d3d0-4870-9b88-8f1f8b7ab77d" (UID: "95995db6-d3d0-4870-9b88-8f1f8b7ab77d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:22:25.024084 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.023977    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-kube-api-access-hcftn" (OuterVolumeSpecName: "kube-api-access-hcftn") pod "95995db6-d3d0-4870-9b88-8f1f8b7ab77d" (UID: "95995db6-d3d0-4870-9b88-8f1f8b7ab77d"). InnerVolumeSpecName "kube-api-access-hcftn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:22:25.024153 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.024115    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "95995db6-d3d0-4870-9b88-8f1f8b7ab77d" (UID: "95995db6-d3d0-4870-9b88-8f1f8b7ab77d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:22:25.024194 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.024150    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "95995db6-d3d0-4870-9b88-8f1f8b7ab77d" (UID: "95995db6-d3d0-4870-9b88-8f1f8b7ab77d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:22:25.058350 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.058325    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:22:25.123081 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.123050    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcftn\" (UniqueName: \"kubernetes.io/projected/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-kube-api-access-hcftn\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.123081 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.123077    2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-ui-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.123081 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.123088    2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-offline-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.123355 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.123098    2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-online-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.123355 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.123107    2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95995db6-d3d0-4870-9b88-8f1f8b7ab77d-feast-data\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.223721 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.223691    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-online-tls\") pod \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") "
May 28 15:22:25.223852 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.223731    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea4a236-5624-40ba-895f-7fdba4a7f70f-feast-data\") pod \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") "
May 28 15:22:25.223852 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.223761    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-registry-tls\") pod \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") "
May 28 15:22:25.223852 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.223787    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ntdk\" (UniqueName: \"kubernetes.io/projected/9ea4a236-5624-40ba-895f-7fdba4a7f70f-kube-api-access-6ntdk\") pod \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") "
May 28 15:22:25.223852 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.223851    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-ui-tls\") pod \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") "
May 28 15:22:25.224004 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.223885    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-offline-tls\") pod \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\" (UID: \"9ea4a236-5624-40ba-895f-7fdba4a7f70f\") "
May 28 15:22:25.224376 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.224343    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea4a236-5624-40ba-895f-7fdba4a7f70f-feast-data" (OuterVolumeSpecName: "feast-data") pod "9ea4a236-5624-40ba-895f-7fdba4a7f70f" (UID: "9ea4a236-5624-40ba-895f-7fdba4a7f70f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:22:25.225927 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.225900    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9ea4a236-5624-40ba-895f-7fdba4a7f70f" (UID: "9ea4a236-5624-40ba-895f-7fdba4a7f70f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:22:25.226030 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.225952    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-online-tls" (OuterVolumeSpecName: "online-tls") pod "9ea4a236-5624-40ba-895f-7fdba4a7f70f" (UID: "9ea4a236-5624-40ba-895f-7fdba4a7f70f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:22:25.226030 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.225960    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "9ea4a236-5624-40ba-895f-7fdba4a7f70f" (UID: "9ea4a236-5624-40ba-895f-7fdba4a7f70f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:22:25.226245 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.226217    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9ea4a236-5624-40ba-895f-7fdba4a7f70f" (UID: "9ea4a236-5624-40ba-895f-7fdba4a7f70f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:22:25.226245 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.226231    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea4a236-5624-40ba-895f-7fdba4a7f70f-kube-api-access-6ntdk" (OuterVolumeSpecName: "kube-api-access-6ntdk") pod "9ea4a236-5624-40ba-895f-7fdba4a7f70f" (UID: "9ea4a236-5624-40ba-895f-7fdba4a7f70f"). InnerVolumeSpecName "kube-api-access-6ntdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:22:25.325083 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.324981    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-registry-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.325083 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.325017    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ntdk\" (UniqueName: \"kubernetes.io/projected/9ea4a236-5624-40ba-895f-7fdba4a7f70f-kube-api-access-6ntdk\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.325083 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.325032    2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-ui-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.325083 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.325043    2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-offline-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.325083 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.325056    2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ea4a236-5624-40ba-895f-7fdba4a7f70f-online-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.325083 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.325066    2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea4a236-5624-40ba-895f-7fdba4a7f70f-feast-data\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:22:25.543721 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.543685    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq" event={"ID":"9ea4a236-5624-40ba-895f-7fdba4a7f70f","Type":"ContainerDied","Data":"14d47ac40efebefa9ebfc995723278094fa44050ab31ca9b4f323d0650a8f1fc"}
May 28 15:22:25.543721 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.543724    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"
May 28 15:22:25.544216 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.543733    2582 scope.go:117] "RemoveContainer" containerID="f34a1c8634262332c9505b82e475d6103574ce3327dde92055586c120192fae7"
May 28 15:22:25.546372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.546345    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g" event={"ID":"95995db6-d3d0-4870-9b88-8f1f8b7ab77d","Type":"ContainerDied","Data":"9c2c5aa94be75adfaf2c0a8cf10456caee1b534b176fea4ad40fd9d18fa3ce48"}
May 28 15:22:25.546372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.546349    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"
May 28 15:22:25.552821 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.552798    2582 scope.go:117] "RemoveContainer" containerID="fda98f76cf080141dca7b9d95c606634e4027b59272e2180289f7c124122b136"
May 28 15:22:25.559923 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.559906    2582 scope.go:117] "RemoveContainer" containerID="737d1420b3e1da86d8484fd6cc9ff12ee7739c535b551a9735157bcbee1aa55d"
May 28 15:22:25.566835 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.566814    2582 scope.go:117] "RemoveContainer" containerID="3092df12e1f1477c653b75c7b813fe7fad007ee5075850c8583761255d30f202"
May 28 15:22:25.567282 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.567261    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"]
May 28 15:22:25.570550 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.570530    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d4d6c99fb-dh5mq"]
May 28 15:22:25.574769 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.574752    2582 scope.go:117] "RemoveContainer" containerID="de0b4c2a87f17e78d34baec46017b429d7b43e71e853aa0be4e22ffa2da36209"
May 28 15:22:25.581056 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.581028    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"]
May 28 15:22:25.582038 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.582025    2582 scope.go:117] "RemoveContainer" containerID="a8fcad80e31d6532fbe40910194afbbe71125fc6c29fbeac796c73082aed92a2"
May 28 15:22:25.584320 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.584301    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b4d59d8bd-9qx2g"]
May 28 15:22:25.589261 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.589245    2582 scope.go:117] "RemoveContainer" containerID="fd21bf53b8f0a7abb01fdcff7c7bbc47d165f09bc2f3fcd4e3cb6f0a6c897d81"
May 28 15:22:25.596034 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.596013    2582 scope.go:117] "RemoveContainer" containerID="af48f1c3daf214a98d1c6a1ab391cfb107de10cb62b306e91e79cb616cb098da"
May 28 15:22:25.602403 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.602375    2582 scope.go:117] "RemoveContainer" containerID="4b9e8c737c38b5725d5a3372ff0dc8af47afa21c7fcde03acc75f6229134b7c0"
May 28 15:22:25.608638 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.608621    2582 scope.go:117] "RemoveContainer" containerID="10cf1a1d36a33d85d7ed320d13ac4dea0a9c9c4765677ed415c6a918e6b0446a"
May 28 15:22:25.614958 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:25.614943    2582 scope.go:117] "RemoveContainer" containerID="984631ade658e189465b350f2cd596ad7ad773718586b55fa5d066530664ae36"
May 28 15:22:26.292409 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:26.292339    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" path="/var/lib/kubelet/pods/95995db6-d3d0-4870-9b88-8f1f8b7ab77d/volumes"
May 28 15:22:26.292878 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:26.292865    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" path="/var/lib/kubelet/pods/9ea4a236-5624-40ba-895f-7fdba4a7f70f/volumes"
May 28 15:22:37.194039 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194004    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-854544c579-wk2pj"]
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194334    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="registry"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194346    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="registry"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194357    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="online"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194363    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="online"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194371    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="ui"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194376    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="ui"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194383    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="feast-apply"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194388    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="feast-apply"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194393    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="offline"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194398    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="offline"
May 28 15:22:37.194404 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194409    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="online"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194414    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="online"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194420    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="feast-init"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194426    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="feast-init"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194435    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="feast-apply"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194440    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="feast-apply"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194445    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="feast-init"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194450    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="feast-init"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194456    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="offline"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194462    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="offline"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194469    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="ui"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194474    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="ui"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194523    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="offline"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194532    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="offline"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194537    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="registry"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194545    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="online"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194551    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="ui"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194558    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea4a236-5624-40ba-895f-7fdba4a7f70f" containerName="online"
May 28 15:22:37.194713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.194564    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="95995db6-d3d0-4870-9b88-8f1f8b7ab77d" containerName="ui"
May 28 15:22:37.197520 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.197504    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-wk2pj"
May 28 15:22:37.200167 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.200143    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 15:22:37.200293 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.200152    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 15:22:37.200293 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.200217    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-txq77\""
May 28 15:22:37.205248 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.205225    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-wk2pj"]
May 28 15:22:37.224275 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.224253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqfk\" (UniqueName: \"kubernetes.io/projected/f7b76305-6755-46e9-af6a-be94155df842-kube-api-access-zzqfk\") pod \"redis-854544c579-wk2pj\" (UID: \"f7b76305-6755-46e9-af6a-be94155df842\") " pod="test-ns-feast/redis-854544c579-wk2pj"
May 28 15:22:37.324830 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.324800    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqfk\" (UniqueName: \"kubernetes.io/projected/f7b76305-6755-46e9-af6a-be94155df842-kube-api-access-zzqfk\") pod \"redis-854544c579-wk2pj\" (UID: \"f7b76305-6755-46e9-af6a-be94155df842\") " pod="test-ns-feast/redis-854544c579-wk2pj"
May 28 15:22:37.332674 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.332645    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqfk\" (UniqueName: \"kubernetes.io/projected/f7b76305-6755-46e9-af6a-be94155df842-kube-api-access-zzqfk\") pod \"redis-854544c579-wk2pj\" (UID: \"f7b76305-6755-46e9-af6a-be94155df842\") " pod="test-ns-feast/redis-854544c579-wk2pj"
May 28 15:22:37.515215 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.515137    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-wk2pj"
May 28 15:22:37.629752 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.629725    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-wk2pj"]
May 28 15:22:37.632807 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:22:37.632783    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7b76305_6755_46e9_af6a_be94155df842.slice/crio-f00ec561312b932497de2ff97dcb4ea8425e72158db9e79c7b15649cfc74b818 WatchSource:0}: Error finding container f00ec561312b932497de2ff97dcb4ea8425e72158db9e79c7b15649cfc74b818: Status 404 returned error can't find the container with id f00ec561312b932497de2ff97dcb4ea8425e72158db9e79c7b15649cfc74b818
May 28 15:22:37.634596 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:37.634578    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 15:22:38.586437 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:38.586404    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-wk2pj" event={"ID":"f7b76305-6755-46e9-af6a-be94155df842","Type":"ContainerStarted","Data":"f00ec561312b932497de2ff97dcb4ea8425e72158db9e79c7b15649cfc74b818"}
May 28 15:22:39.590005 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:39.589963    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-wk2pj" event={"ID":"f7b76305-6755-46e9-af6a-be94155df842","Type":"ContainerStarted","Data":"62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0"}
May 28 15:22:39.605292 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:39.605249    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/redis-854544c579-wk2pj" podStartSLOduration=1.15353686 podStartE2EDuration="2.605237956s" podCreationTimestamp="2026-05-28 15:22:37 +0000 UTC" firstStartedPulling="2026-05-28 15:22:37.634707708 +0000 UTC m=+709.933777198" lastFinishedPulling="2026-05-28 15:22:39.086408789 +0000 UTC m=+711.385478294" observedRunningTime="2026-05-28 15:22:39.603882553 +0000 UTC m=+711.902952063" watchObservedRunningTime="2026-05-28 15:22:39.605237956 +0000 UTC m=+711.904307466"
May 28 15:22:47.888273 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:47.888236    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"]
May 28 15:22:47.892461 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:47.892440    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:47.894656 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:47.894634    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 28 15:22:47.895048 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:47.894782    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 28 15:22:47.895048 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:47.894830    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 28 15:22:47.895048 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:47.894912    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 28 15:22:47.895048 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:47.894922    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-vzcsg\""
May 28 15:22:47.895048 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:47.895048    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 28 15:22:47.900414 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:47.900379    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"]
May 28 15:22:48.007755 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.007713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-online-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.007957 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.007932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmc82\" (UniqueName: \"kubernetes.io/projected/d507fe88-bdce-4db9-b120-ca47e22ad921-kube-api-access-dmc82\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.008046 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.008013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-offline-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.008103 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.008080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d507fe88-bdce-4db9-b120-ca47e22ad921-feast-data\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.008182 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.008139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-registry-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.108539 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.108506    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-offline-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.108698 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.108554    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d507fe88-bdce-4db9-b120-ca47e22ad921-feast-data\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.108698 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.108652    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-registry-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.108698 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:22:48.108667    2582 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found
May 28 15:22:48.108831 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.108699    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-online-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.108831 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:22:48.108734    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-offline-tls podName:d507fe88-bdce-4db9-b120-ca47e22ad921 nodeName:}" failed. No retries permitted until 2026-05-28 15:22:48.608712933 +0000 UTC m=+720.907782445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-offline-tls") pod "feast-credit-scoring-7d6ffb8f79-5qns2" (UID: "d507fe88-bdce-4db9-b120-ca47e22ad921") : secret "feast-credit-scoring-offline-tls" not found
May 28 15:22:48.108831 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.108781    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dmc82\" (UniqueName: \"kubernetes.io/projected/d507fe88-bdce-4db9-b120-ca47e22ad921-kube-api-access-dmc82\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.109032 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.109010    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d507fe88-bdce-4db9-b120-ca47e22ad921-feast-data\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.111252 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.111233    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-online-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.111372 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.111353    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-registry-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.117270 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.117245    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmc82\" (UniqueName: \"kubernetes.io/projected/d507fe88-bdce-4db9-b120-ca47e22ad921-kube-api-access-dmc82\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.614507 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.614478    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-offline-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.616914 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.616882    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-offline-tls\") pod \"feast-credit-scoring-7d6ffb8f79-5qns2\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") " pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.806292 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.806255    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-vzcsg\""
May 28 15:22:48.814283 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.814256    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:48.936870 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:48.936846    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"]
May 28 15:22:48.938923 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:22:48.938895    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd507fe88_bdce_4db9_b120_ca47e22ad921.slice/crio-2b2537a1a40c452c8f3b51b489465001b03e1494ea2b0e712ebc31634f63e55d WatchSource:0}: Error finding container 2b2537a1a40c452c8f3b51b489465001b03e1494ea2b0e712ebc31634f63e55d: Status 404 returned error can't find the container with id 2b2537a1a40c452c8f3b51b489465001b03e1494ea2b0e712ebc31634f63e55d
May 28 15:22:49.620499 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:49.620459    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerStarted","Data":"c7c4db852c18975cdb34a4bc4d242bfc9fe2f4f62679a5ad805cdbd684ad00b7"}
May 28 15:22:49.620499 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:49.620493    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerStarted","Data":"2b2537a1a40c452c8f3b51b489465001b03e1494ea2b0e712ebc31634f63e55d"}
May 28 15:22:50.624917 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:50.624880    2582 generic.go:358] "Generic (PLEG): container finished" podID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerID="c7c4db852c18975cdb34a4bc4d242bfc9fe2f4f62679a5ad805cdbd684ad00b7" exitCode=0
May 28 15:22:50.625420 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:50.624943    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerDied","Data":"c7c4db852c18975cdb34a4bc4d242bfc9fe2f4f62679a5ad805cdbd684ad00b7"}
May 28 15:22:50.629522 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:50.629502    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 28 15:22:51.630487 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:51.630456    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerStarted","Data":"b2b2d39899eb0bb3901f75c82678cdc9988de6b75538159539a6a10873ab9763"}
May 28 15:22:57.649321 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:57.649285    2582 generic.go:358] "Generic (PLEG): container finished" podID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerID="b2b2d39899eb0bb3901f75c82678cdc9988de6b75538159539a6a10873ab9763" exitCode=0
May 28 15:22:57.649677 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:57.649356    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerDied","Data":"b2b2d39899eb0bb3901f75c82678cdc9988de6b75538159539a6a10873ab9763"}
May 28 15:22:57.653189 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:57.653173    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 28 15:22:58.657387 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:58.657350    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerStarted","Data":"024c2eac32a0c6d78cc47ed6596357311d4a4c4cde598da2a17082f6abe679ae"}
May 28 15:22:58.657387 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:58.657393    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerStarted","Data":"d8790a630a4ccb7cf70abfd8a219571300aad76458b1f4fcbc559c4164f7ac98"}
May 28 15:22:58.657903 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:58.657406    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerStarted","Data":"1a62f330e8919324728c44a299d510709a8d94a67fccbd17c6ee690b2339ee6a"}
May 28 15:22:58.686221 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:58.686166    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podStartSLOduration=11.686148015 podStartE2EDuration="11.686148015s" podCreationTimestamp="2026-05-28 15:22:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:22:58.684322043 +0000 UTC m=+730.983391553" watchObservedRunningTime="2026-05-28 15:22:58.686148015 +0000 UTC m=+730.985217526"
May 28 15:22:58.814858 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:58.814824    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:58.814858 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:58.814864    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:22:58.815108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:22:58.814873    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:23:00.814986 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:00.814952    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:23:00.815446 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:00.815020    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:23:00.815446 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:00.815035    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:23:00.816550 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:00.816520    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused"
May 28 15:23:00.816705 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:00.816590    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 15:23:00.816705 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:00.816525    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused"
May 28 15:23:03.815047 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:03.815000    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused"
May 28 15:23:03.815719 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:03.815697    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:23:03.816605 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:03.816585    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:23:03.820855 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:03.820834    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:23:04.683540 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:04.683505    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:23:06.814612 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:06.814565    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused"
May 28 15:23:09.815403 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:09.815368    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:23:09.816274 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:23:09.816251    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:24:10.345066 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.345031    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"]
May 28 15:24:10.345526 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.345378    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="registry" containerID="cri-o://1a62f330e8919324728c44a299d510709a8d94a67fccbd17c6ee690b2339ee6a" gracePeriod=30
May 28 15:24:10.345526 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.345436    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="online" containerID="cri-o://d8790a630a4ccb7cf70abfd8a219571300aad76458b1f4fcbc559c4164f7ac98" gracePeriod=30
May 28 15:24:10.345526 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.345448    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="offline" containerID="cri-o://024c2eac32a0c6d78cc47ed6596357311d4a4c4cde598da2a17082f6abe679ae" gracePeriod=30
May 28 15:24:10.368164 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.368115    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-wk2pj"]
May 28 15:24:10.368364 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.368332    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/redis-854544c579-wk2pj" podUID="f7b76305-6755-46e9-af6a-be94155df842" containerName="redis" containerID="cri-o://62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0" gracePeriod=30
May 28 15:24:10.608820 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.608799    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-wk2pj"
May 28 15:24:10.712491 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.712456    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzqfk\" (UniqueName: \"kubernetes.io/projected/f7b76305-6755-46e9-af6a-be94155df842-kube-api-access-zzqfk\") pod \"f7b76305-6755-46e9-af6a-be94155df842\" (UID: \"f7b76305-6755-46e9-af6a-be94155df842\") "
May 28 15:24:10.714504 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.714452    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b76305-6755-46e9-af6a-be94155df842-kube-api-access-zzqfk" (OuterVolumeSpecName: "kube-api-access-zzqfk") pod "f7b76305-6755-46e9-af6a-be94155df842" (UID: "f7b76305-6755-46e9-af6a-be94155df842"). InnerVolumeSpecName "kube-api-access-zzqfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:24:10.813929 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.813885    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzqfk\" (UniqueName: \"kubernetes.io/projected/f7b76305-6755-46e9-af6a-be94155df842-kube-api-access-zzqfk\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:24:10.892203 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.892172    2582 generic.go:358] "Generic (PLEG): container finished" podID="f7b76305-6755-46e9-af6a-be94155df842" containerID="62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0" exitCode=0
May 28 15:24:10.892376 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.892233    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-wk2pj"
May 28 15:24:10.892376 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.892268    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-wk2pj" event={"ID":"f7b76305-6755-46e9-af6a-be94155df842","Type":"ContainerDied","Data":"62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0"}
May 28 15:24:10.892376 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.892303    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-wk2pj" event={"ID":"f7b76305-6755-46e9-af6a-be94155df842","Type":"ContainerDied","Data":"f00ec561312b932497de2ff97dcb4ea8425e72158db9e79c7b15649cfc74b818"}
May 28 15:24:10.892376 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.892318    2582 scope.go:117] "RemoveContainer" containerID="62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0"
May 28 15:24:10.899817 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.899798    2582 scope.go:117] "RemoveContainer" containerID="62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0"
May 28 15:24:10.900080 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:24:10.900062    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0\": container with ID starting with 62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0 not found: ID does not exist" containerID="62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0"
May 28 15:24:10.900168 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.900088    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0"} err="failed to get container status \"62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0\": rpc error: code = NotFound desc = could not find container \"62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0\": container with ID starting with 62c9dc91472fc4b2dbd46666a1aca4c1738b83ea8e37605187e295497aff72d0 not found: ID does not exist"
May 28 15:24:10.914089 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.914059    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-wk2pj"]
May 28 15:24:10.917214 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:10.917194    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-854544c579-wk2pj"]
May 28 15:24:11.898187 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:11.898159    2582 generic.go:358] "Generic (PLEG): container finished" podID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerID="d8790a630a4ccb7cf70abfd8a219571300aad76458b1f4fcbc559c4164f7ac98" exitCode=0
May 28 15:24:11.898187 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:11.898191    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerDied","Data":"d8790a630a4ccb7cf70abfd8a219571300aad76458b1f4fcbc559c4164f7ac98"}
May 28 15:24:12.292078 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:12.292002    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b76305-6755-46e9-af6a-be94155df842" path="/var/lib/kubelet/pods/f7b76305-6755-46e9-af6a-be94155df842/volumes"
May 28 15:24:14.679696 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:14.679656    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 15:24:24.679892 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:24.679849    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 15:24:34.679329 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:34.679289    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 15:24:34.679794 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:34.679389    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:24:40.985915 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:40.985879    2582 generic.go:358] "Generic (PLEG): container finished" podID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerID="024c2eac32a0c6d78cc47ed6596357311d4a4c4cde598da2a17082f6abe679ae" exitCode=137
May 28 15:24:40.985915 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:40.985914    2582 generic.go:358] "Generic (PLEG): container finished" podID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerID="1a62f330e8919324728c44a299d510709a8d94a67fccbd17c6ee690b2339ee6a" exitCode=137
May 28 15:24:40.986250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:40.985945    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerDied","Data":"024c2eac32a0c6d78cc47ed6596357311d4a4c4cde598da2a17082f6abe679ae"}
May 28 15:24:40.986250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:40.985981    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerDied","Data":"1a62f330e8919324728c44a299d510709a8d94a67fccbd17c6ee690b2339ee6a"}
May 28 15:24:40.986250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:40.985995    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2" event={"ID":"d507fe88-bdce-4db9-b120-ca47e22ad921","Type":"ContainerDied","Data":"2b2537a1a40c452c8f3b51b489465001b03e1494ea2b0e712ebc31634f63e55d"}
May 28 15:24:40.986250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:40.986003    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b2537a1a40c452c8f3b51b489465001b03e1494ea2b0e712ebc31634f63e55d"
May 28 15:24:40.988903 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:40.988887    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:24:41.049743 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.049715    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmc82\" (UniqueName: \"kubernetes.io/projected/d507fe88-bdce-4db9-b120-ca47e22ad921-kube-api-access-dmc82\") pod \"d507fe88-bdce-4db9-b120-ca47e22ad921\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") "
May 28 15:24:41.049904 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.049772    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-offline-tls\") pod \"d507fe88-bdce-4db9-b120-ca47e22ad921\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") "
May 28 15:24:41.049904 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.049789    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d507fe88-bdce-4db9-b120-ca47e22ad921-feast-data\") pod \"d507fe88-bdce-4db9-b120-ca47e22ad921\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") "
May 28 15:24:41.049904 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.049816    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-registry-tls\") pod \"d507fe88-bdce-4db9-b120-ca47e22ad921\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") "
May 28 15:24:41.049904 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.049851    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-online-tls\") pod \"d507fe88-bdce-4db9-b120-ca47e22ad921\" (UID: \"d507fe88-bdce-4db9-b120-ca47e22ad921\") "
May 28 15:24:41.051897 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.051868    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d507fe88-bdce-4db9-b120-ca47e22ad921" (UID: "d507fe88-bdce-4db9-b120-ca47e22ad921"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:24:41.051996 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.051918    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-online-tls" (OuterVolumeSpecName: "online-tls") pod "d507fe88-bdce-4db9-b120-ca47e22ad921" (UID: "d507fe88-bdce-4db9-b120-ca47e22ad921"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:24:41.051996 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.051951    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d507fe88-bdce-4db9-b120-ca47e22ad921" (UID: "d507fe88-bdce-4db9-b120-ca47e22ad921"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:24:41.052089 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.052075    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d507fe88-bdce-4db9-b120-ca47e22ad921-kube-api-access-dmc82" (OuterVolumeSpecName: "kube-api-access-dmc82") pod "d507fe88-bdce-4db9-b120-ca47e22ad921" (UID: "d507fe88-bdce-4db9-b120-ca47e22ad921"). InnerVolumeSpecName "kube-api-access-dmc82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:24:41.058263 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.058242    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d507fe88-bdce-4db9-b120-ca47e22ad921-feast-data" (OuterVolumeSpecName: "feast-data") pod "d507fe88-bdce-4db9-b120-ca47e22ad921" (UID: "d507fe88-bdce-4db9-b120-ca47e22ad921"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:24:41.150795 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.150738    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmc82\" (UniqueName: \"kubernetes.io/projected/d507fe88-bdce-4db9-b120-ca47e22ad921-kube-api-access-dmc82\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:24:41.150795 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.150763    2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-offline-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:24:41.150795 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.150771    2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d507fe88-bdce-4db9-b120-ca47e22ad921-feast-data\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:24:41.150795 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.150779    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-registry-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:24:41.150795 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.150787    2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d507fe88-bdce-4db9-b120-ca47e22ad921-online-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:24:41.989352 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:41.989322    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"
May 28 15:24:42.010137 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:42.010091    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"]
May 28 15:24:42.013177 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:42.013156    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d6ffb8f79-5qns2"]
May 28 15:24:42.292083 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:24:42.292010    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" path="/var/lib/kubelet/pods/d507fe88-bdce-4db9-b120-ca47e22ad921/volumes"
May 28 15:25:48.265320 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:25:48.265294    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:25:48.272019 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:25:48.271998    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:26:26.956108 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956072    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"]
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956414    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b76305-6755-46e9-af6a-be94155df842" containerName="redis"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956428    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b76305-6755-46e9-af6a-be94155df842" containerName="redis"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956443    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="feast-init"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956448    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="feast-init"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956466    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="feast-apply"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956474    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="feast-apply"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956481    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="online"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956486    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="online"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956493    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="offline"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956498    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="offline"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956507    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="registry"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956512    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="registry"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956563    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="online"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956572    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="registry"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956580    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="d507fe88-bdce-4db9-b120-ca47e22ad921" containerName="offline"
May 28 15:26:26.956622 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.956590    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="f7b76305-6755-46e9-af6a-be94155df842" containerName="redis"
May 28 15:26:26.959743 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.959725    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:26.962643 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.962621    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 28 15:26:26.962801 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.962683    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-5wzgc\""
May 28 15:26:26.962801 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.962688    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 28 15:26:26.962801 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.962785    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 28 15:26:26.962965 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.962822    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 28 15:26:26.962965 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.962932    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 28 15:26:26.962965 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.962946    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 28 15:26:26.970334 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:26.970314    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"]
May 28 15:26:27.060585 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.060546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz49l\" (UniqueName: \"kubernetes.io/projected/0b1f4265-a139-44aa-9772-52a53450f620-kube-api-access-lz49l\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.060747 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.060605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1f4265-a139-44aa-9772-52a53450f620-feast-data\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.060747 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.060677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-registry-tls\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.060747 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.060704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-online-tls\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.161959 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.161926    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lz49l\" (UniqueName: \"kubernetes.io/projected/0b1f4265-a139-44aa-9772-52a53450f620-kube-api-access-lz49l\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.162104 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.161977    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1f4265-a139-44aa-9772-52a53450f620-feast-data\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.162204 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.162177    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-registry-tls\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.162266 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.162215    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-online-tls\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.162336 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.162321    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1f4265-a139-44aa-9772-52a53450f620-feast-data\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.164585 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.164566    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-registry-tls\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.164684 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.164594    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-online-tls\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.169495 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.169474    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz49l\" (UniqueName: \"kubernetes.io/projected/0b1f4265-a139-44aa-9772-52a53450f620-kube-api-access-lz49l\") pod \"feast-credit-scoring-586588d879-kffgs\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") " pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.270274 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.270188    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:27.392729 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:27.392704    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"]
May 28 15:26:27.395687 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:26:27.395651    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1f4265_a139_44aa_9772_52a53450f620.slice/crio-c3d494752573e07c66bbcf8638df72eb82936643fedc2780e44293aeda0ee662 WatchSource:0}: Error finding container c3d494752573e07c66bbcf8638df72eb82936643fedc2780e44293aeda0ee662: Status 404 returned error can't find the container with id c3d494752573e07c66bbcf8638df72eb82936643fedc2780e44293aeda0ee662
May 28 15:26:28.293526 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:28.293489    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerStarted","Data":"7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27"}
May 28 15:26:28.293882 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:28.293531    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerStarted","Data":"c3d494752573e07c66bbcf8638df72eb82936643fedc2780e44293aeda0ee662"}
May 28 15:26:29.298175 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:29.298146    2582 generic.go:358] "Generic (PLEG): container finished" podID="0b1f4265-a139-44aa-9772-52a53450f620" containerID="7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27" exitCode=0
May 28 15:26:29.298175 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:29.298180    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerDied","Data":"7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27"}
May 28 15:26:30.302057 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:30.302016    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerStarted","Data":"582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2"}
May 28 15:26:37.326457 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:37.326418    2582 generic.go:358] "Generic (PLEG): container finished" podID="0b1f4265-a139-44aa-9772-52a53450f620" containerID="582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2" exitCode=0
May 28 15:26:37.326832 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:37.326490    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerDied","Data":"582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2"}
May 28 15:26:38.332590 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:38.332550    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerStarted","Data":"f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1"}
May 28 15:26:38.332960 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:38.332603    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerStarted","Data":"f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c"}
May 28 15:26:38.354006 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:38.353957    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podStartSLOduration=12.353942967 podStartE2EDuration="12.353942967s" podCreationTimestamp="2026-05-28 15:26:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:26:38.351780443 +0000 UTC m=+950.650849994" watchObservedRunningTime="2026-05-28 15:26:38.353942967 +0000 UTC m=+950.653012478"
May 28 15:26:39.270863 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:39.270819    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:39.270863 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:39.270862    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:39.272393 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:39.272360    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 15:26:39.272393 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:39.272376    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused"
May 28 15:26:42.271364 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:42.271312    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused"
May 28 15:26:42.271842 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:42.271409    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 15:26:45.271211 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:45.271168    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused"
May 28 15:26:45.278021 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:45.277996    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:45.278468 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:45.278443    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:45.281343 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:45.281323    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:47.270773 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:47.270731    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:48.271427 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:48.271398    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:48.367144 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:48.367096    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:26:52.874351 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.874316    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"]
May 28 15:26:52.879078 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.879061    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:52.881523 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.881501    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 28 15:26:52.881523 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.881502    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 28 15:26:52.881671 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.881526    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 28 15:26:52.881671 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.881655    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-87q8d\""
May 28 15:26:52.887408 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.887337    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"]
May 28 15:26:52.980412 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.980383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llp9\" (UniqueName: \"kubernetes.io/projected/5438792d-9b7f-4cb8-915e-6faf231e4305-kube-api-access-5llp9\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:52.980569 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.980426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5438792d-9b7f-4cb8-915e-6faf231e4305-feast-data\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:52.980569 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.980478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-registry-tls\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:52.980569 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:52.980524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-online-tls\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.081110 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.081087    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5llp9\" (UniqueName: \"kubernetes.io/projected/5438792d-9b7f-4cb8-915e-6faf231e4305-kube-api-access-5llp9\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.081272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.081134    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5438792d-9b7f-4cb8-915e-6faf231e4305-feast-data\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.081272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.081155    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-registry-tls\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.081272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.081177    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-online-tls\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.081530 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.081511    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5438792d-9b7f-4cb8-915e-6faf231e4305-feast-data\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.083538 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.083521    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-registry-tls\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.083598 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.083566    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-online-tls\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.088331 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.088311    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llp9\" (UniqueName: \"kubernetes.io/projected/5438792d-9b7f-4cb8-915e-6faf231e4305-kube-api-access-5llp9\") pod \"feast-driver-ranking-6d64c68ffb-xq99f\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") " pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.191370 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.191300    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:26:53.311367 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.311340    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"]
May 28 15:26:53.313634 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:26:53.313605    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5438792d_9b7f_4cb8_915e_6faf231e4305.slice/crio-5687d6239d17731c82d04b9458a085b2632bcd74eb13552dd7d4a97d60fa6567 WatchSource:0}: Error finding container 5687d6239d17731c82d04b9458a085b2632bcd74eb13552dd7d4a97d60fa6567: Status 404 returned error can't find the container with id 5687d6239d17731c82d04b9458a085b2632bcd74eb13552dd7d4a97d60fa6567
May 28 15:26:53.382652 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.382623    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerStarted","Data":"82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858"}
May 28 15:26:53.382741 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:53.382658    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerStarted","Data":"5687d6239d17731c82d04b9458a085b2632bcd74eb13552dd7d4a97d60fa6567"}
May 28 15:26:58.398036 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:58.398005    2582 generic.go:358] "Generic (PLEG): container finished" podID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerID="82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858" exitCode=0
May 28 15:26:58.398448 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:58.398054    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerDied","Data":"82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858"}
May 28 15:26:59.403257 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:26:59.403221    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerStarted","Data":"c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485"}
May 28 15:27:05.425587 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:05.425553    2582 generic.go:358] "Generic (PLEG): container finished" podID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerID="c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485" exitCode=0
May 28 15:27:05.425947 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:05.425623    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerDied","Data":"c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485"}
May 28 15:27:06.430976 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:06.430945    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerStarted","Data":"f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318"}
May 28 15:27:06.431393 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:06.430984    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerStarted","Data":"b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2"}
May 28 15:27:06.451628 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:06.451585    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podStartSLOduration=14.451570323 podStartE2EDuration="14.451570323s" podCreationTimestamp="2026-05-28 15:26:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:27:06.449062737 +0000 UTC m=+978.748132247" watchObservedRunningTime="2026-05-28 15:27:06.451570323 +0000 UTC m=+978.750639833"
May 28 15:27:08.191713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:08.191661    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:27:08.191713 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:08.191712    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:27:08.193201 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:08.193168    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused"
May 28 15:27:08.193331 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:08.193204    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused"
May 28 15:27:11.192272 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:11.192218    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused"
May 28 15:27:11.438065 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:11.438027    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:27:11.438566 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:11.438545    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:27:11.441299 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:11.441275    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:27:13.191743 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:13.191707    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:27:14.192251 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:14.192199    2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused"
May 28 15:27:17.192588 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:17.192558    2582 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:27:17.193493 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:27:17.193472    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:28:30.817474 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:30.817389    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"]
May 28 15:28:30.818016 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:30.817777    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="registry" containerID="cri-o://f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c" gracePeriod=30
May 28 15:28:30.818016 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:30.817842    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="online" containerID="cri-o://f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1" gracePeriod=30
May 28 15:28:30.823777 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:30.823753    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"]
May 28 15:28:30.824033 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:30.823996    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="registry" containerID="cri-o://b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2" gracePeriod=30
May 28 15:28:30.824090 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:30.824040    2582 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="online" containerID="cri-o://f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318" gracePeriod=30
May 28 15:28:31.439475 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:31.439432    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused"
May 28 15:28:31.696368 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:31.696295    2582 generic.go:358] "Generic (PLEG): container finished" podID="0b1f4265-a139-44aa-9772-52a53450f620" containerID="f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1" exitCode=0
May 28 15:28:31.696492 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:31.696365    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerDied","Data":"f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1"}
May 28 15:28:31.698257 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:31.698235    2582 generic.go:358] "Generic (PLEG): container finished" podID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerID="f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318" exitCode=0
May 28 15:28:31.698359 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:31.698300    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerDied","Data":"f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318"}
May 28 15:28:35.279445 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:35.279406    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 15:28:41.439476 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:41.439434    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused"
May 28 15:28:45.279503 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:45.279465    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 15:28:51.438755 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:51.438710    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused"
May 28 15:28:51.439155 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:51.438820    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:28:55.278831 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:55.278790    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 15:28:55.279245 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:28:55.278944    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:29:01.477696 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.477675    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:29:01.480723 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.480707    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:29:01.569020 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.568986    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-online-tls\") pod \"0b1f4265-a139-44aa-9772-52a53450f620\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") "
May 28 15:29:01.569020 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.569031    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-online-tls\") pod \"5438792d-9b7f-4cb8-915e-6faf231e4305\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") "
May 28 15:29:01.569279 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.569055    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz49l\" (UniqueName: \"kubernetes.io/projected/0b1f4265-a139-44aa-9772-52a53450f620-kube-api-access-lz49l\") pod \"0b1f4265-a139-44aa-9772-52a53450f620\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") "
May 28 15:29:01.569279 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.569196    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5438792d-9b7f-4cb8-915e-6faf231e4305-feast-data\") pod \"5438792d-9b7f-4cb8-915e-6faf231e4305\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") "
May 28 15:29:01.569279 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.569254    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5llp9\" (UniqueName: \"kubernetes.io/projected/5438792d-9b7f-4cb8-915e-6faf231e4305-kube-api-access-5llp9\") pod \"5438792d-9b7f-4cb8-915e-6faf231e4305\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") "
May 28 15:29:01.569428 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.569312    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-registry-tls\") pod \"0b1f4265-a139-44aa-9772-52a53450f620\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") "
May 28 15:29:01.569428 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.569348    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1f4265-a139-44aa-9772-52a53450f620-feast-data\") pod \"0b1f4265-a139-44aa-9772-52a53450f620\" (UID: \"0b1f4265-a139-44aa-9772-52a53450f620\") "
May 28 15:29:01.569428 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.569402    2582 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-registry-tls\") pod \"5438792d-9b7f-4cb8-915e-6faf231e4305\" (UID: \"5438792d-9b7f-4cb8-915e-6faf231e4305\") "
May 28 15:29:01.569961 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.569731    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5438792d-9b7f-4cb8-915e-6faf231e4305-feast-data" (OuterVolumeSpecName: "feast-data") pod "5438792d-9b7f-4cb8-915e-6faf231e4305" (UID: "5438792d-9b7f-4cb8-915e-6faf231e4305"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:29:01.571692 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.571655    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0b1f4265-a139-44aa-9772-52a53450f620" (UID: "0b1f4265-a139-44aa-9772-52a53450f620"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:29:01.571867 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.571835    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5438792d-9b7f-4cb8-915e-6faf231e4305" (UID: "5438792d-9b7f-4cb8-915e-6faf231e4305"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:29:01.571867 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.571850    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5438792d-9b7f-4cb8-915e-6faf231e4305-kube-api-access-5llp9" (OuterVolumeSpecName: "kube-api-access-5llp9") pod "5438792d-9b7f-4cb8-915e-6faf231e4305" (UID: "5438792d-9b7f-4cb8-915e-6faf231e4305"). InnerVolumeSpecName "kube-api-access-5llp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:29:01.571867 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.571859    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1f4265-a139-44aa-9772-52a53450f620-kube-api-access-lz49l" (OuterVolumeSpecName: "kube-api-access-lz49l") pod "0b1f4265-a139-44aa-9772-52a53450f620" (UID: "0b1f4265-a139-44aa-9772-52a53450f620"). InnerVolumeSpecName "kube-api-access-lz49l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 15:29:01.572046 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.571959    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-online-tls" (OuterVolumeSpecName: "online-tls") pod "0b1f4265-a139-44aa-9772-52a53450f620" (UID: "0b1f4265-a139-44aa-9772-52a53450f620"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:29:01.572594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.572574    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-online-tls" (OuterVolumeSpecName: "online-tls") pod "5438792d-9b7f-4cb8-915e-6faf231e4305" (UID: "5438792d-9b7f-4cb8-915e-6faf231e4305"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 15:29:01.579964 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.579941    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1f4265-a139-44aa-9772-52a53450f620-feast-data" (OuterVolumeSpecName: "feast-data") pod "0b1f4265-a139-44aa-9772-52a53450f620" (UID: "0b1f4265-a139-44aa-9772-52a53450f620"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 15:29:01.670628 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.670596    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz49l\" (UniqueName: \"kubernetes.io/projected/0b1f4265-a139-44aa-9772-52a53450f620-kube-api-access-lz49l\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:29:01.670628 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.670623    2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5438792d-9b7f-4cb8-915e-6faf231e4305-feast-data\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:29:01.670628 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.670632    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5llp9\" (UniqueName: \"kubernetes.io/projected/5438792d-9b7f-4cb8-915e-6faf231e4305-kube-api-access-5llp9\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:29:01.670844 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.670640    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-registry-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:29:01.670844 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.670650    2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1f4265-a139-44aa-9772-52a53450f620-feast-data\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:29:01.670844 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.670658    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-registry-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:29:01.670844 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.670665    2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b1f4265-a139-44aa-9772-52a53450f620-online-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:29:01.670844 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.670673    2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5438792d-9b7f-4cb8-915e-6faf231e4305-online-tls\") on node \"ip-10-0-130-203.ec2.internal\" DevicePath \"\""
May 28 15:29:01.793413 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.793381    2582 generic.go:358] "Generic (PLEG): container finished" podID="0b1f4265-a139-44aa-9772-52a53450f620" containerID="f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c" exitCode=137
May 28 15:29:01.793581 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.793454    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"
May 28 15:29:01.793581 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.793458    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerDied","Data":"f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c"}
May 28 15:29:01.793581 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.793496    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs" event={"ID":"0b1f4265-a139-44aa-9772-52a53450f620","Type":"ContainerDied","Data":"c3d494752573e07c66bbcf8638df72eb82936643fedc2780e44293aeda0ee662"}
May 28 15:29:01.793581 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.793512    2582 scope.go:117] "RemoveContainer" containerID="f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1"
May 28 15:29:01.795586 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.795558    2582 generic.go:358] "Generic (PLEG): container finished" podID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerID="b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2" exitCode=137
May 28 15:29:01.795717 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.795586    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerDied","Data":"b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2"}
May 28 15:29:01.795717 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.795610    2582 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"
May 28 15:29:01.795717 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.795612    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" event={"ID":"5438792d-9b7f-4cb8-915e-6faf231e4305","Type":"ContainerDied","Data":"5687d6239d17731c82d04b9458a085b2632bcd74eb13552dd7d4a97d60fa6567"}
May 28 15:29:01.802591 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.802538    2582 scope.go:117] "RemoveContainer" containerID="f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c"
May 28 15:29:01.810105 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.810090    2582 scope.go:117] "RemoveContainer" containerID="582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2"
May 28 15:29:01.816777 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.816755    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"]
May 28 15:29:01.817059 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.817042    2582 scope.go:117] "RemoveContainer" containerID="7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27"
May 28 15:29:01.820410 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.820387    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-586588d879-kffgs"]
May 28 15:29:01.824730 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.824519    2582 scope.go:117] "RemoveContainer" containerID="f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1"
May 28 15:29:01.824814 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:29:01.824790    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1\": container with ID starting with f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1 not found: ID does not exist" containerID="f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1"
May 28 15:29:01.824869 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.824824    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1"} err="failed to get container status \"f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1\": rpc error: code = NotFound desc = could not find container \"f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1\": container with ID starting with f3ee51d78906efe8ffec989d5beecbb17b98d5c8f8f55563c36310adedeab4c1 not found: ID does not exist"
May 28 15:29:01.824869 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.824848    2582 scope.go:117] "RemoveContainer" containerID="f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c"
May 28 15:29:01.825098 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:29:01.825082    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c\": container with ID starting with f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c not found: ID does not exist" containerID="f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c"
May 28 15:29:01.825160 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.825105    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c"} err="failed to get container status \"f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c\": rpc error: code = NotFound desc = could not find container \"f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c\": container with ID starting with f28fe490e1f2c83f7e1dd6ef235b25cee6e2835e450c06253de551a4ac94916c not found: ID does not exist"
May 28 15:29:01.825160 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.825139    2582 scope.go:117] "RemoveContainer" containerID="582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2"
May 28 15:29:01.825385 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:29:01.825367    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2\": container with ID starting with 582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2 not found: ID does not exist" containerID="582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2"
May 28 15:29:01.825424 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.825390    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2"} err="failed to get container status \"582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2\": rpc error: code = NotFound desc = could not find container \"582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2\": container with ID starting with 582872ce48d54f91f8310819dcbbe69c71a5687c8647c1df832376c6a7089fd2 not found: ID does not exist"
May 28 15:29:01.825424 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.825406    2582 scope.go:117] "RemoveContainer" containerID="7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27"
May 28 15:29:01.825598 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:29:01.825584    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27\": container with ID starting with 7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27 not found: ID does not exist" containerID="7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27"
May 28 15:29:01.825649 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.825601    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27"} err="failed to get container status \"7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27\": rpc error: code = NotFound desc = could not find container \"7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27\": container with ID starting with 7f25a0b1cf1b92660cdc2c626963ae218d10ae9d40f3e4fdedd493ec1b44dc27 not found: ID does not exist"
May 28 15:29:01.825649 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.825614    2582 scope.go:117] "RemoveContainer" containerID="f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318"
May 28 15:29:01.829362 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.829345    2582 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"]
May 28 15:29:01.832107 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.832087    2582 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f"]
May 28 15:29:01.835989 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.835974    2582 scope.go:117] "RemoveContainer" containerID="b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2"
May 28 15:29:01.842690 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.842675    2582 scope.go:117] "RemoveContainer" containerID="c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485"
May 28 15:29:01.849152 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.849114    2582 scope.go:117] "RemoveContainer" containerID="82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858"
May 28 15:29:01.855414 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.855399    2582 scope.go:117] "RemoveContainer" containerID="f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318"
May 28 15:29:01.855644 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:29:01.855627    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318\": container with ID starting with f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318 not found: ID does not exist" containerID="f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318"
May 28 15:29:01.855709 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.855653    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318"} err="failed to get container status \"f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318\": rpc error: code = NotFound desc = could not find container \"f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318\": container with ID starting with f589091e3d176abfb47a33bad44e5aee2c378a3434d252b0fd214cf62a997318 not found: ID does not exist"
May 28 15:29:01.855709 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.855679    2582 scope.go:117] "RemoveContainer" containerID="b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2"
May 28 15:29:01.855899 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:29:01.855880    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2\": container with ID starting with b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2 not found: ID does not exist" containerID="b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2"
May 28 15:29:01.855944 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.855905    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2"} err="failed to get container status \"b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2\": rpc error: code = NotFound desc = could not find container \"b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2\": container with ID starting with b745f08c945a50491286eb02190d12da4766d2711e7fe7adbc0980db24ceb1a2 not found: ID does not exist"
May 28 15:29:01.855944 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.855920    2582 scope.go:117] "RemoveContainer" containerID="c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485"
May 28 15:29:01.856115 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:29:01.856099    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485\": container with ID starting with c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485 not found: ID does not exist" containerID="c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485"
May 28 15:29:01.856115 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.856130    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485"} err="failed to get container status \"c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485\": rpc error: code = NotFound desc = could not find container \"c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485\": container with ID starting with c5f15843a767c60e9f69737329d401ecb26b401b2a2cfa6b401db8b7d982f485 not found: ID does not exist"
May 28 15:29:01.856115 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.856144    2582 scope.go:117] "RemoveContainer" containerID="82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858"
May 28 15:29:01.856339 ip-10-0-130-203 kubenswrapper[2582]: E0528 15:29:01.856319    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858\": container with ID starting with 82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858 not found: ID does not exist" containerID="82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858"
May 28 15:29:01.856390 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:01.856349    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858"} err="failed to get container status \"82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858\": rpc error: code = NotFound desc = could not find container \"82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858\": container with ID starting with 82bd589b0bf35aa33262693052be1ba74f31776111dd9c28473a89ab28a75858 not found: ID does not exist"
May 28 15:29:02.291427 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:02.291397    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1f4265-a139-44aa-9772-52a53450f620" path="/var/lib/kubelet/pods/0b1f4265-a139-44aa-9772-52a53450f620/volumes"
May 28 15:29:02.291894 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:02.291880    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" path="/var/lib/kubelet/pods/5438792d-9b7f-4cb8-915e-6faf231e4305/volumes"
May 28 15:29:02.440340 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:02.440286    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6d64c68ffb-xq99f" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
May 28 15:29:48.310862 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:48.310799    2582 scope.go:117] "RemoveContainer" containerID="b2b2d39899eb0bb3901f75c82678cdc9988de6b75538159539a6a10873ab9763"
May 28 15:29:48.318130 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:48.318100    2582 scope.go:117] "RemoveContainer" containerID="024c2eac32a0c6d78cc47ed6596357311d4a4c4cde598da2a17082f6abe679ae"
May 28 15:29:48.325056 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:48.325038    2582 scope.go:117] "RemoveContainer" containerID="1a62f330e8919324728c44a299d510709a8d94a67fccbd17c6ee690b2339ee6a"
May 28 15:29:48.331752 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:48.331736    2582 scope.go:117] "RemoveContainer" containerID="c7c4db852c18975cdb34a4bc4d242bfc9fe2f4f62679a5ad805cdbd684ad00b7"
May 28 15:29:48.338055 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:29:48.338036    2582 scope.go:117] "RemoveContainer" containerID="d8790a630a4ccb7cf70abfd8a219571300aad76458b1f4fcbc559c4164f7ac98"
May 28 15:30:48.288038 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:30:48.288006    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:30:48.295340 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:30:48.295318    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:35:48.313506 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:35:48.313478    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:35:48.320470 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:35:48.320444    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:39:37.509995 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.509960    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lc5n8/must-gather-vz6xs"]
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510285    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="feast-apply"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510296    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="feast-apply"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510305    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="online"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510312    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="online"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510321    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="online"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510327    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="online"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510335    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="registry"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510341    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="registry"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510348    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="registry"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510353    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="registry"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510361    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="feast-init"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510366    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="feast-init"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510372    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="feast-init"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510376    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="feast-init"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510386    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="feast-apply"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510390    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="feast-apply"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510435    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="online"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510443    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="registry"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510451    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b1f4265-a139-44aa-9772-52a53450f620" containerName="registry"
May 28 15:39:37.510496 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.510457    2582 memory_manager.go:356] "RemoveStaleState removing state" podUID="5438792d-9b7f-4cb8-915e-6faf231e4305" containerName="online"
May 28 15:39:37.513414 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.513398    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lc5n8/must-gather-vz6xs"
May 28 15:39:37.515965 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.515942    2582 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lc5n8\"/\"default-dockercfg-45tb4\""
May 28 15:39:37.516098 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.515963    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lc5n8\"/\"openshift-service-ca.crt\""
May 28 15:39:37.516098 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.516023    2582 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lc5n8\"/\"kube-root-ca.crt\""
May 28 15:39:37.521058 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.521036    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lc5n8/must-gather-vz6xs"]
May 28 15:39:37.701271 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.701234    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvrs\" (UniqueName: \"kubernetes.io/projected/59251875-8e6e-439a-84b6-0322c042d45d-kube-api-access-qrvrs\") pod \"must-gather-vz6xs\" (UID: \"59251875-8e6e-439a-84b6-0322c042d45d\") " pod="openshift-must-gather-lc5n8/must-gather-vz6xs"
May 28 15:39:37.701271 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.701271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59251875-8e6e-439a-84b6-0322c042d45d-must-gather-output\") pod \"must-gather-vz6xs\" (UID: \"59251875-8e6e-439a-84b6-0322c042d45d\") " pod="openshift-must-gather-lc5n8/must-gather-vz6xs"
May 28 15:39:37.801721 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.801625    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvrs\" (UniqueName: \"kubernetes.io/projected/59251875-8e6e-439a-84b6-0322c042d45d-kube-api-access-qrvrs\") pod \"must-gather-vz6xs\" (UID: \"59251875-8e6e-439a-84b6-0322c042d45d\") " pod="openshift-must-gather-lc5n8/must-gather-vz6xs"
May 28 15:39:37.801721 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.801665    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59251875-8e6e-439a-84b6-0322c042d45d-must-gather-output\") pod \"must-gather-vz6xs\" (UID: \"59251875-8e6e-439a-84b6-0322c042d45d\") " pod="openshift-must-gather-lc5n8/must-gather-vz6xs"
May 28 15:39:37.801971 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.801950    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59251875-8e6e-439a-84b6-0322c042d45d-must-gather-output\") pod \"must-gather-vz6xs\" (UID: \"59251875-8e6e-439a-84b6-0322c042d45d\") " pod="openshift-must-gather-lc5n8/must-gather-vz6xs"
May 28 15:39:37.809065 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.809038    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvrs\" (UniqueName: \"kubernetes.io/projected/59251875-8e6e-439a-84b6-0322c042d45d-kube-api-access-qrvrs\") pod \"must-gather-vz6xs\" (UID: \"59251875-8e6e-439a-84b6-0322c042d45d\") " pod="openshift-must-gather-lc5n8/must-gather-vz6xs"
May 28 15:39:37.839864 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.839841    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lc5n8/must-gather-vz6xs"
May 28 15:39:37.960951 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.960919    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lc5n8/must-gather-vz6xs"]
May 28 15:39:37.964220 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:39:37.964193    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59251875_8e6e_439a_84b6_0322c042d45d.slice/crio-2b2f95f8a4d2510f39ba254dc08126f9002ebaab36612e80846252077a892860 WatchSource:0}: Error finding container 2b2f95f8a4d2510f39ba254dc08126f9002ebaab36612e80846252077a892860: Status 404 returned error can't find the container with id 2b2f95f8a4d2510f39ba254dc08126f9002ebaab36612e80846252077a892860
May 28 15:39:37.966110 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:37.966089    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 15:39:38.617377 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:38.617341    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc5n8/must-gather-vz6xs" event={"ID":"59251875-8e6e-439a-84b6-0322c042d45d","Type":"ContainerStarted","Data":"2b2f95f8a4d2510f39ba254dc08126f9002ebaab36612e80846252077a892860"}
May 28 15:39:39.623387 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:39.623348    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc5n8/must-gather-vz6xs" event={"ID":"59251875-8e6e-439a-84b6-0322c042d45d","Type":"ContainerStarted","Data":"8eb85ba821d3439171d5e28885b7ca64f02ca29669fa1a7ac2748b0689cdcc36"}
May 28 15:39:39.623387 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:39.623393    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc5n8/must-gather-vz6xs" event={"ID":"59251875-8e6e-439a-84b6-0322c042d45d","Type":"ContainerStarted","Data":"e708eb60e02007d9d0157d10fa9929562b474faf28dd894bcefe860c081a9813"}
May 28 15:39:39.638332 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:39.638264    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-lc5n8/must-gather-vz6xs" podStartSLOduration=1.770087888 podStartE2EDuration="2.638245203s" podCreationTimestamp="2026-05-28 15:39:37 +0000 UTC" firstStartedPulling="2026-05-28 15:39:37.966289121 +0000 UTC m=+1730.265358618" lastFinishedPulling="2026-05-28 15:39:38.834446439 +0000 UTC m=+1731.133515933" observedRunningTime="2026-05-28 15:39:39.637679358 +0000 UTC m=+1731.936748869" watchObservedRunningTime="2026-05-28 15:39:39.638245203 +0000 UTC m=+1731.937314716"
May 28 15:39:40.105858 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:40.105767    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jlddv_61cd75b2-d230-4aab-bfe8-8b8727742281/global-pull-secret-syncer/0.log"
May 28 15:39:40.154056 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:40.154000    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9pvmp_dc64f60e-b2fb-46c2-b608-93ed763fbdc6/konnectivity-agent/0.log"
May 28 15:39:40.231992 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:40.231954    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-203.ec2.internal_1691e4b38b2da49454ca8518fd1a04cb/haproxy/0.log"
May 28 15:39:42.967994 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:42.967941    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9857b26e-d11e-42ed-ab8f-69828b9ca00b/alertmanager/0.log"
May 28 15:39:43.003053 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.003018    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9857b26e-d11e-42ed-ab8f-69828b9ca00b/config-reloader/0.log"
May 28 15:39:43.035415 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.035299    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9857b26e-d11e-42ed-ab8f-69828b9ca00b/kube-rbac-proxy-web/0.log"
May 28 15:39:43.063134 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.063095    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9857b26e-d11e-42ed-ab8f-69828b9ca00b/kube-rbac-proxy/0.log"
May 28 15:39:43.095646 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.095616    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9857b26e-d11e-42ed-ab8f-69828b9ca00b/kube-rbac-proxy-metric/0.log"
May 28 15:39:43.126437 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.126357    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9857b26e-d11e-42ed-ab8f-69828b9ca00b/prom-label-proxy/0.log"
May 28 15:39:43.149493 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.149452    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9857b26e-d11e-42ed-ab8f-69828b9ca00b/init-config-reloader/0.log"
May 28 15:39:43.288635 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.288553    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b99c6c9bf-l6sp2_64bef1cf-1067-4750-a373-48a51752bc00/metrics-server/0.log"
May 28 15:39:43.348984 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.348948    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kkd7p_a131cb29-6526-475f-b894-1bcedb93b59a/node-exporter/0.log"
May 28 15:39:43.374103 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.374077    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kkd7p_a131cb29-6526-475f-b894-1bcedb93b59a/kube-rbac-proxy/0.log"
May 28 15:39:43.396151 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.396105    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kkd7p_a131cb29-6526-475f-b894-1bcedb93b59a/init-textfile/0.log"
May 28 15:39:43.640712 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.640651    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a26bc17d-d3e4-483e-b945-9384c9b63980/prometheus/0.log"
May 28 15:39:43.660875 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.660834    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a26bc17d-d3e4-483e-b945-9384c9b63980/config-reloader/0.log"
May 28 15:39:43.682152 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.682109    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a26bc17d-d3e4-483e-b945-9384c9b63980/thanos-sidecar/0.log"
May 28 15:39:43.701576 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.701547    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a26bc17d-d3e4-483e-b945-9384c9b63980/kube-rbac-proxy-web/0.log"
May 28 15:39:43.724141 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.724101    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a26bc17d-d3e4-483e-b945-9384c9b63980/kube-rbac-proxy/0.log"
May 28 15:39:43.746487 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.746454    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a26bc17d-d3e4-483e-b945-9384c9b63980/kube-rbac-proxy-thanos/0.log"
May 28 15:39:43.766475 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.766442    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a26bc17d-d3e4-483e-b945-9384c9b63980/init-config-reloader/0.log"
May 28 15:39:43.795110 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.795051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-4tbcl_b80c2e55-3fb2-4a4e-bcd7-d59323fd6224/prometheus-operator/0.log"
May 28 15:39:43.812748 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.812708    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-4tbcl_b80c2e55-3fb2-4a4e-bcd7-d59323fd6224/kube-rbac-proxy/0.log"
May 28 15:39:43.838828 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.838797    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-2kl74_41ec3631-f453-4883-8bb9-5954671f1a73/prometheus-operator-admission-webhook/0.log"
May 28 15:39:43.942745 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.942667    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb8859d8-bbq8s_3c8ad83a-578d-4851-bbdf-d839d3f2ef34/thanos-query/0.log"
May 28 15:39:43.964667 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.964606    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb8859d8-bbq8s_3c8ad83a-578d-4851-bbdf-d839d3f2ef34/kube-rbac-proxy-web/0.log"
May 28 15:39:43.985645 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:43.985614    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb8859d8-bbq8s_3c8ad83a-578d-4851-bbdf-d839d3f2ef34/kube-rbac-proxy/0.log"
May 28 15:39:44.005989 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:44.005949    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb8859d8-bbq8s_3c8ad83a-578d-4851-bbdf-d839d3f2ef34/prom-label-proxy/0.log"
May 28 15:39:44.026217 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:44.026179    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb8859d8-bbq8s_3c8ad83a-578d-4851-bbdf-d839d3f2ef34/kube-rbac-proxy-rules/0.log"
May 28 15:39:44.046833 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:44.046803    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb8859d8-bbq8s_3c8ad83a-578d-4851-bbdf-d839d3f2ef34/kube-rbac-proxy-metrics/0.log"
May 28 15:39:45.555228 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:45.555157    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/2.log"
May 28 15:39:45.559538 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:45.559513    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-k6lc4_d9bc276c-5d34-4b0e-83ea-a5ed8af32e8d/console-operator/3.log"
May 28 15:39:46.567400 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.567365    2582 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"]
May 28 15:39:46.572352 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.572320    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.578467 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.578434    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"]
May 28 15:39:46.679343 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.679307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-sys\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.679522 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.679404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-proc\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.679522 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.679440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscl5\" (UniqueName: \"kubernetes.io/projected/0871dc16-2e9e-4116-ab71-954829bfbd18-kube-api-access-kscl5\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.679522 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.679470    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-lib-modules\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.679522 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.679500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-podres\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.780980 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.780934    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-proc\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.781195 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.781004    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kscl5\" (UniqueName: \"kubernetes.io/projected/0871dc16-2e9e-4116-ab71-954829bfbd18-kube-api-access-kscl5\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.781195 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.781039    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-lib-modules\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.781195 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.781077    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-podres\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.781195 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.781143    2582 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-sys\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.781444 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.781265    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-sys\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.781444 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.781331    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-proc\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.781658 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.781637    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-podres\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.781705 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.781636    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0871dc16-2e9e-4116-ab71-954829bfbd18-lib-modules\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.788965 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.788938    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscl5\" (UniqueName: \"kubernetes.io/projected/0871dc16-2e9e-4116-ab71-954829bfbd18-kube-api-access-kscl5\") pod \"perf-node-gather-daemonset-jpjxj\" (UID: \"0871dc16-2e9e-4116-ab71-954829bfbd18\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.881736 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.881708    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9n5pb_713bf691-c64b-485f-b8f8-7094daac71eb/dns/0.log"
May 28 15:39:46.886642 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.886619    2582 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:46.900493 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:46.900467    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9n5pb_713bf691-c64b-485f-b8f8-7094daac71eb/kube-rbac-proxy/0.log"
May 28 15:39:47.008396 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:47.008349    2582 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"]
May 28 15:39:47.011806 ip-10-0-130-203 kubenswrapper[2582]: W0528 15:39:47.011782    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0871dc16_2e9e_4116_ab71_954829bfbd18.slice/crio-0ee976dcb280ba96f455fa875e0e38a50173e23c7213c29ed6c6059a193d9473 WatchSource:0}: Error finding container 0ee976dcb280ba96f455fa875e0e38a50173e23c7213c29ed6c6059a193d9473: Status 404 returned error can't find the container with id 0ee976dcb280ba96f455fa875e0e38a50173e23c7213c29ed6c6059a193d9473
May 28 15:39:47.014938 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:47.014918    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sd8jw_f11838e8-88b0-4035-85e4-99fd5e9026c3/dns-node-resolver/0.log"
May 28 15:39:47.404010 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:47.403927    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjtv8_2fe05b8d-d2bb-40c3-9642-61b447da82ab/node-ca/0.log"
May 28 15:39:47.657250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:47.657154    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj" event={"ID":"0871dc16-2e9e-4116-ab71-954829bfbd18","Type":"ContainerStarted","Data":"d8ca08b358c2f348bf1fba973a5a3f13f94e8d07abeaf7bc04ddb2d8553cfd56"}
May 28 15:39:47.657250 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:47.657196    2582 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj" event={"ID":"0871dc16-2e9e-4116-ab71-954829bfbd18","Type":"ContainerStarted","Data":"0ee976dcb280ba96f455fa875e0e38a50173e23c7213c29ed6c6059a193d9473"}
May 28 15:39:47.657750 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:47.657281    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:47.671801 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:47.671726    2582 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj" podStartSLOduration=1.671705138 podStartE2EDuration="1.671705138s" podCreationTimestamp="2026-05-28 15:39:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:39:47.671360442 +0000 UTC m=+1739.970429953" watchObservedRunningTime="2026-05-28 15:39:47.671705138 +0000 UTC m=+1739.970774653"
May 28 15:39:48.353764 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:48.353736    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5lwgs_37fbb52e-9da0-488d-b0a2-8706e14120a8/serve-healthcheck-canary/0.log"
May 28 15:39:48.820744 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:48.820714    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vg8zf_cfab63fc-ed9e-4984-b0f7-e0afd45b74bf/kube-rbac-proxy/0.log"
May 28 15:39:48.838633 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:48.838609    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vg8zf_cfab63fc-ed9e-4984-b0f7-e0afd45b74bf/exporter/0.log"
May 28 15:39:48.856751 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:48.856707    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vg8zf_cfab63fc-ed9e-4984-b0f7-e0afd45b74bf/extractor/0.log"
May 28 15:39:52.485053 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:52.485022    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-85lk6_979677a1-5c8c-4286-aa13-3d92aab72251/migrator/0.log"
May 28 15:39:52.502887 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:52.502860    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-85lk6_979677a1-5c8c-4286-aa13-3d92aab72251/graceful-termination/0.log"
May 28 15:39:53.664703 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:53.664678    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpq57_2ccdd5a3-8460-4c56-863d-239d9c2da33f/kube-multus-additional-cni-plugins/0.log"
May 28 15:39:53.669504 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:53.669481    2582 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-jpjxj"
May 28 15:39:53.684817 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:53.684794    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpq57_2ccdd5a3-8460-4c56-863d-239d9c2da33f/egress-router-binary-copy/0.log"
May 28 15:39:53.704674 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:53.704649    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpq57_2ccdd5a3-8460-4c56-863d-239d9c2da33f/cni-plugins/0.log"
May 28 15:39:53.723394 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:53.723369    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpq57_2ccdd5a3-8460-4c56-863d-239d9c2da33f/bond-cni-plugin/0.log"
May 28 15:39:53.741662 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:53.741636    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpq57_2ccdd5a3-8460-4c56-863d-239d9c2da33f/routeoverride-cni/0.log"
May 28 15:39:53.760448 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:53.760423    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpq57_2ccdd5a3-8460-4c56-863d-239d9c2da33f/whereabouts-cni-bincopy/0.log"
May 28 15:39:53.779067 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:53.779042    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpq57_2ccdd5a3-8460-4c56-863d-239d9c2da33f/whereabouts-cni/0.log"
May 28 15:39:53.957192 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:53.957106    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-b9mt2_2f261580-fa49-4bdd-bffa-1dc49eb17669/kube-multus/0.log"
May 28 15:39:54.020796 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:54.020765    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9lnwm_60775ca8-b4ea-4176-bda3-58ad85bbc806/network-metrics-daemon/0.log"
May 28 15:39:54.038973 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:54.038949    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9lnwm_60775ca8-b4ea-4176-bda3-58ad85bbc806/kube-rbac-proxy/0.log"
May 28 15:39:55.277112 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:55.277082    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tx2n2_3d8b93da-b2de-4a1c-b264-604d7329745f/ovn-controller/0.log"
May 28 15:39:55.304920 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:55.304873    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tx2n2_3d8b93da-b2de-4a1c-b264-604d7329745f/ovn-acl-logging/0.log"
May 28 15:39:55.324749 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:55.324712    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tx2n2_3d8b93da-b2de-4a1c-b264-604d7329745f/kube-rbac-proxy-node/0.log"
May 28 15:39:55.347509 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:55.347485    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tx2n2_3d8b93da-b2de-4a1c-b264-604d7329745f/kube-rbac-proxy-ovn-metrics/0.log"
May 28 15:39:55.365310 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:55.365280    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tx2n2_3d8b93da-b2de-4a1c-b264-604d7329745f/northd/0.log"
May 28 15:39:55.384594 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:55.384548    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tx2n2_3d8b93da-b2de-4a1c-b264-604d7329745f/nbdb/0.log"
May 28 15:39:55.406210 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:55.406143    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tx2n2_3d8b93da-b2de-4a1c-b264-604d7329745f/sbdb/0.log"
May 28 15:39:55.524397 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:55.524322    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tx2n2_3d8b93da-b2de-4a1c-b264-604d7329745f/ovnkube-controller/0.log"
May 28 15:39:56.560280 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:56.560206    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-j5kk5_9884c711-a48a-419c-abc1-d50cf28d0b46/network-check-target-container/0.log"
May 28 15:39:57.475988 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:57.475963    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qd8qz_085514bb-7b92-43e7-b961-c3d99f6cd3a5/iptables-alerter/0.log"
May 28 15:39:57.995138 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:57.995102    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-dcgvr_81941efe-61e4-45de-9f42-87037c076e64/tuned/0.log"
May 28 15:39:59.485993 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:59.485965    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-29nc7_47d63f1b-668d-473b-a657-3bf3a3ac0a49/cluster-samples-operator/0.log"
May 28 15:39:59.500032 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:39:59.500010    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-29nc7_47d63f1b-668d-473b-a657-3bf3a3ac0a49/cluster-samples-operator-watch/0.log"
May 28 15:40:00.600233 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:40:00.600206    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-67d667c7d-kqzcs_eea18e20-42f1-488e-a0a5-72af0baebc09/service-ca-controller/0.log"
May 28 15:40:00.976418 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:40:00.976392    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-hczrb_95c19bfa-ba5d-4068-8cf3-1e76902be2a5/csi-driver/0.log"
May 28 15:40:00.996714 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:40:00.996690    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-hczrb_95c19bfa-ba5d-4068-8cf3-1e76902be2a5/csi-node-driver-registrar/0.log"
May 28 15:40:01.014525 ip-10-0-130-203 kubenswrapper[2582]: I0528 15:40:01.014498    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-hczrb_95c19bfa-ba5d-4068-8cf3-1e76902be2a5/csi-liveness-probe/0.log"