May 27 23:31:05.724332 ip-10-0-130-189 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 27 23:31:05.724341 ip-10-0-130-189 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 27 23:31:05.724348 ip-10-0-130-189 systemd[1]: kubelet.service: Failed with result 'resources'.
May 27 23:31:05.724564 ip-10-0-130-189 systemd[1]: Failed to start Kubernetes Kubelet.
May 27 23:31:15.724709 ip-10-0-130-189 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 27 23:31:15.724726 ip-10-0-130-189 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 76d60a5b15164efd85ce4de21ac3fd6e --
May 27 23:33:29.211806 ip-10-0-130-189 systemd[1]: Starting Kubernetes Kubelet...
May 27 23:33:29.715400 ip-10-0-130-189 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 27 23:33:29.715400 ip-10-0-130-189 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 27 23:33:29.715400 ip-10-0-130-189 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 27 23:33:29.715400 ip-10-0-130-189 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 27 23:33:29.715400 ip-10-0-130-189 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 27 23:33:29.717342 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.717181    2562 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 27 23:33:29.722673 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722652    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:33:29.722673 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722669    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:33:29.722673 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722674    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:33:29.722673 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722677    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:33:29.722673 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722681    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722685    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722691    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722695    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722699    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722703    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722707    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722710    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722714    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722717    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722721    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722724    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722728    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722732    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722735    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722739    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722743    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722747    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722751    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722754    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:33:29.722983 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722759    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722762    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722779    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722783    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722787    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722791    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722795    2562 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722799    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722803    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722806    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722810    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722814    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722818    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722822    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722827    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722831    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722836    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722840    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722844    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722848    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:33:29.723795 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722853    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722857    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722861    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722866    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722870    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722875    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722879    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722883    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722887    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722891    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722898    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722902    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722905    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722910    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722914    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722918    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722923    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722927    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722933    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722939    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:33:29.724677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722944    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722949    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722953    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722957    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722961    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722966    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722972    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722979    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722984    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722990    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.722994    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723000    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723006    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723010    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723015    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723020    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723024    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723028    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723033    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:33:29.725557 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723037    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723041    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723045    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723696    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723707    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723711    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723715    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723719    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723724    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723729    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723733    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723737    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723741    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723746    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723750    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723755    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723758    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723763    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723767    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723772    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:33:29.726072 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723776    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723780    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723785    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723789    2562 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723793    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723797    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723802    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723806    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723812    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723818    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723823    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723828    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723832    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723837    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723841    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723845    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723850    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723854    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723858    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723863    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:33:29.726654 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723867    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723871    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723875    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723879    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723884    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723888    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723892    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723896    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723900    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723904    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723908    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723913    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723918    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723922    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723926    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723930    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723934    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723938    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723947    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723954    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:33:29.727212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723958    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723963    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723967    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723971    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723976    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723981    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723985    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723989    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723994    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.723998    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724002    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724006    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724010    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724014    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724019    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724023    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724027    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724031    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724035    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:33:29.727894 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724039    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724044    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724048    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724056    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724063    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724068    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724073    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724078    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724083    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.724088    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724190    2562 flags.go:64] FLAG: --address="0.0.0.0"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724206    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724217    2562 flags.go:64] FLAG: --anonymous-auth="true"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724223    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724229    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724235    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724241    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724248    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724254    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724259    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724265    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 27 23:33:29.728440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724270    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724275    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724280    2562 flags.go:64] FLAG: --cgroup-root=""
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724286    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724290    2562 flags.go:64] FLAG: --client-ca-file=""
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724295    2562 flags.go:64] FLAG: --cloud-config=""
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724300    2562 flags.go:64] FLAG: --cloud-provider="external"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724305    2562 flags.go:64] FLAG: --cluster-dns="[]"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724312    2562 flags.go:64] FLAG: --cluster-domain=""
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724317    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724322    2562 flags.go:64] FLAG: --config-dir=""
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724327    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724333    2562 flags.go:64] FLAG: --container-log-max-files="5"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724339    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724344    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724349    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724354    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724360    2562 flags.go:64] FLAG: --contention-profiling="false"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724365    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724370    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724376    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724381    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724388    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724393    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724398    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 27 23:33:29.728973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724403    2562 flags.go:64] FLAG: --enable-load-reader="false"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724408    2562 flags.go:64] FLAG: --enable-server="true"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724413    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724420    2562 flags.go:64] FLAG: --event-burst="100"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724425    2562 flags.go:64] FLAG: --event-qps="50"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724430    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724435    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724440    2562 flags.go:64] FLAG: --eviction-hard=""
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724446    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724451    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724457    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724462    2562 flags.go:64] FLAG: --eviction-soft=""
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724466    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724471    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724476    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724481    2562 flags.go:64] FLAG: --experimental-mounter-path=""
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724485    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724490    2562 flags.go:64] FLAG: --fail-swap-on="true"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724495    2562 flags.go:64] FLAG: --feature-gates=""
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724501    2562 flags.go:64] FLAG: --file-check-frequency="20s"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724506    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724512    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724517    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724522    2562 flags.go:64] FLAG: --healthz-port="10248"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724542    2562 flags.go:64] FLAG: --help="false"
May 27 23:33:29.729759 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724547    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-130-189.ec2.internal"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724553    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724557    2562 flags.go:64] FLAG: --http-check-frequency="20s"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724562    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724568    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724574    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724579    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724586    2562 flags.go:64] FLAG: --image-service-endpoint=""
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724590    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724595    2562 flags.go:64] FLAG: --kube-api-burst="100"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724600    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724606    2562 flags.go:64] FLAG: --kube-api-qps="50"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724611    2562 flags.go:64] FLAG: --kube-reserved=""
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724616    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724621    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724626    2562 flags.go:64] FLAG: --kubelet-cgroups=""
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724631    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724635    2562 flags.go:64] FLAG: --lock-file=""
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724640    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724646    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724651    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724660    2562 flags.go:64] FLAG: --log-json-split-stream="false"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724665    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724670    2562 flags.go:64] FLAG: --log-text-split-stream="false"
May 27 23:33:29.730391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724675    2562 flags.go:64] FLAG: --logging-format="text"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724680    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724686    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724691    2562 flags.go:64] FLAG: --manifest-url=""
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724696    2562 flags.go:64] FLAG: --manifest-url-header=""
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724703    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724709    2562 flags.go:64] FLAG: --max-open-files="1000000"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724716    2562 flags.go:64] FLAG: --max-pods="110"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724721    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724726    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724731    2562 flags.go:64] FLAG: --memory-manager-policy="None"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724736    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724741    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724746    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724751    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724763    2562 flags.go:64] FLAG: --node-status-max-images="50"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724769    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724774    2562 flags.go:64] FLAG: --oom-score-adj="-999"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724779    2562 flags.go:64] FLAG: --pod-cidr=""
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724784    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724793    2562 flags.go:64] FLAG: --pod-manifest-path=""
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724798    2562 flags.go:64] FLAG: --pod-max-pids="-1"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724803    2562 flags.go:64] FLAG: --pods-per-core="0"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724808    2562 flags.go:64] FLAG: --port="10250"
May 27 23:33:29.731020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724813    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724817    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02ab797c46639da2b"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724823    2562 flags.go:64] FLAG: --qos-reserved=""
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724828    2562 flags.go:64] FLAG: --read-only-port="10255"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724832    2562 flags.go:64] FLAG: --register-node="true"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724837    2562 flags.go:64] FLAG: --register-schedulable="true"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724842    2562 flags.go:64] FLAG: --register-with-taints=""
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724849    2562 flags.go:64] FLAG: --registry-burst="10"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724854    2562 flags.go:64] FLAG: --registry-qps="5"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724862    2562 flags.go:64] FLAG: --reserved-cpus=""
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724867    2562 flags.go:64] FLAG: --reserved-memory=""
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724874    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724878    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724883    2562 flags.go:64] FLAG: --rotate-certificates="false"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724888    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724893    2562 flags.go:64] FLAG: --runonce="false"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724897    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724903    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724909    2562 flags.go:64] FLAG: --seccomp-default="false"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724914    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724919    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724924    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724930    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724935    2562 flags.go:64] FLAG: --storage-driver-password="root"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724940    2562 flags.go:64] FLAG: --storage-driver-secure="false"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724945    2562 flags.go:64] FLAG: --storage-driver-table="stats"
May 27 23:33:29.731633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724950    2562 flags.go:64] FLAG: --storage-driver-user="root"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724955    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724960    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724965    2562 flags.go:64] FLAG: --system-cgroups=""
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724969    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724978    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724983    2562 flags.go:64] FLAG: --tls-cert-file=""
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724988    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.724996    2562 flags.go:64] FLAG: --tls-min-version=""
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.725000    2562 flags.go:64] FLAG: --tls-private-key-file=""
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.725004    2562 flags.go:64] FLAG: --topology-manager-policy="none"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.725009    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.725014    2562 flags.go:64] FLAG: --topology-manager-scope="container"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.725019    2562 flags.go:64] FLAG: --v="2"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.725026    2562 flags.go:64] FLAG: --version="false"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.725034    2562 flags.go:64] FLAG: --vmodule=""
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.725042    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.725047    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725206    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725212    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725217    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725222    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725226    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:33:29.732290 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725231    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725236    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725240    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725245    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725250    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725255    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725259    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725263    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725267    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725273    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725279    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725285    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725289    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725294    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725299    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725304    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725308    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725313    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725317    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:33:29.732967 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725321    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725327    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725333    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725339    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725343    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725349    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725358    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725362    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725366    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725371    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725375    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725380    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725384    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725388    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725392    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725397    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725400    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725405    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725409    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:33:29.733453 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725414    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725418    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725422    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725427    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725433    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725437    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725442    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725446    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725450    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725453    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725458    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725462    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725466    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725471    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725475    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725479    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725483    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725487    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725493    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:33:29.733958 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725499    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725503    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725507    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725511    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725515    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725519    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725523    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725545    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725550    2562 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725553    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725557    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725561    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725565    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725569    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725574    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725577    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725582    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725586    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725591    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725596    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:33:29.734434 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725600    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:33:29.734947 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725605    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:33:29.734947 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725609    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:33:29.734947 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.725613    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:33:29.734947 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.726594    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 27 23:33:29.735688 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.735668    2562 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 27 23:33:29.735725 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.735691    2562 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 27 23:33:29.735756 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735746    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:33:29.735756 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735752    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:33:29.735756 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735756    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735761    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735765    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735770    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735774    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735777    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735779    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735782    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735785    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735787    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735789    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735792    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735795    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735798    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735801    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735803    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735806    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735810    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735813    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735815    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:33:29.735837 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735818    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735820    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735823    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735825    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735828    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735831    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735833    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735837    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735842    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735848    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735852    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735854    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735858    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735860    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735863    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735865    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735868    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735870    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735873    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:33:29.736359 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735875    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735878    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735880    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735883    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735885    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735888    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735890    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735892    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735895    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735898    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735900    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735902    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735905    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735907    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735910    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735914    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735918    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735923    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735928    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:33:29.736857 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735933    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735936    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735939    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735942    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735946    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735949    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735952    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735955    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735957    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735960    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735963    2562 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735965    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735968    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735970    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735973    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735975    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735978    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735980    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735983    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735985    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:33:29.737331 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735988    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735991    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.735995    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736000    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736003    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736008    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.736014    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736135    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736143    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736148    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736154    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736158    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736163    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736167    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736170    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:33:29.737901 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736173    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736176    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736179    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736182    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736185    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736187    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736190    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736192    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736195    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736197    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736200    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736202    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736205    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736208    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736210    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736212    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736215    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736217    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736220    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:33:29.738285 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736224    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736230    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736235    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736239    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736242    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736244    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736247    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736251    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736253    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736256    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736258    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736261    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736263    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736266    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736269    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736271    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736274    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736277    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736279    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736282    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:33:29.738781 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736284    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736287    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736289    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736292    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736295    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736297    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736301    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736304    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736308    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736312    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736316    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736319    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736321    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736324    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736326    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736328    2562 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736331    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736333    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736336    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736339    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:33:29.739288 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736342    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736344    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736347    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736349    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736352    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736354    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736357    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736359    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736362    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736365    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736368    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736371    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736373    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736376    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736380    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736383    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736388    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736392    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:33:29.739799 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:29.736395    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:33:29.740246 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.736400    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 27 23:33:29.740246 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.737250    2562 server.go:962] "Client rotation is on, will bootstrap in background"
May 27 23:33:29.740454 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.740440    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 27 23:33:29.741534 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.741512    2562 server.go:1019] "Starting client certificate rotation"
May 27 23:33:29.741647 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.741630    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 27 23:33:29.741698 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.741686    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 27 23:33:29.770233 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.770203    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 27 23:33:29.775497 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.775469    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 27 23:33:29.791122 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.791096    2562 log.go:25] "Validated CRI v1 runtime API"
May 27 23:33:29.797541 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.797507    2562 log.go:25] "Validated CRI v1 image API"
May 27 23:33:29.798963 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.798939    2562 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 27 23:33:29.801499 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.801475    2562 fs.go:135] Filesystem UUIDs: map[03677645-f701-45c6-83f1-c3e1da27b1fc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a916d508-d97a-4cca-b699-baf2efdabc96:/dev/nvme0n1p3]
May 27 23:33:29.801609 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.801500    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 27 23:33:29.804774 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.804752    2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 27 23:33:29.805973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.805855    2562 manager.go:217] Machine: {Timestamp:2026-05-27 23:33:29.805346545 +0000 UTC m=+0.463209011 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100102 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f4e5859f15c883f8e72268fe483a SystemUUID:ec24f4e5-859f-15c8-83f8-e72268fe483a BootID:76d60a5b-1516-4efd-85ce-4de21ac3fd6e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:44:c0:ae:c2:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:44:c0:ae:c2:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:cc:d6:15:8a:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 27 23:33:29.805973 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.805965    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 27 23:33:29.806145 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.806127    2562 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 27 23:33:29.809294 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.809266    2562 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 27 23:33:29.809472 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.809297    2562 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-189.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 27 23:33:29.809567 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.809486    2562 topology_manager.go:138] "Creating topology manager with none policy"
May 27 23:33:29.809567 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.809499    2562 container_manager_linux.go:306] "Creating device plugin manager"
May 27 23:33:29.809567 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.809510    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 27 23:33:29.809567 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.809548    2562 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 27 23:33:29.812001 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.811987    2562 state_mem.go:36] "Initialized new in-memory state store"
May 27 23:33:29.812141 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.812129    2562 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 27 23:33:29.814714 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.814702    2562 kubelet.go:491] "Attempting to sync node with API server"
May 27 23:33:29.814780 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.814724    2562 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 27 23:33:29.814780 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.814741    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 27 23:33:29.814780 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.814754    2562 kubelet.go:397] "Adding apiserver pod source"
May 27 23:33:29.814780 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.814766    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 27 23:33:29.815969 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.815955    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 27 23:33:29.816056 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.815978    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 27 23:33:29.820006 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.819991    2562 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 27 23:33:29.822523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.822510    2562 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 27 23:33:29.823509 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823495    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 27 23:33:29.823584 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823517    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 27 23:33:29.823584 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823542    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 27 23:33:29.823584 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823550    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 27 23:33:29.823584 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823560    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 27 23:33:29.823584 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823568    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 27 23:33:29.823584 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823575    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 27 23:33:29.823584 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823583    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 27 23:33:29.823832 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823593    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 27 23:33:29.823832 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823601    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 27 23:33:29.823832 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823611    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 27 23:33:29.823927 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823917    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 27 23:33:29.823967 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823946    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 27 23:33:29.823967 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.823952    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 27 23:33:29.827561 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.827548    2562 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 27 23:33:29.827638 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.827602    2562 server.go:1295] "Started kubelet"
May 27 23:33:29.827713 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.827658    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 27 23:33:29.829982 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.829659    2562 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 27 23:33:29.830084 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.830053    2562 server_v1.go:47] "podresources" method="list" useActivePods=true
May 27 23:33:29.830347 ip-10-0-130-189 systemd[1]: Started Kubernetes Kubelet.
May 27 23:33:29.832082 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.832058    2562 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-189.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 27 23:33:29.832361 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.832331    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-189.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 27 23:33:29.832603 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.832551    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 27 23:33:29.834169 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.834139    2562 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 27 23:33:29.834577 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.834563    2562 server.go:317] "Adding debug handlers to kubelet server"
May 27 23:33:29.840095 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.838998    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-189.ec2.internal.18b3902f20ad2808  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-189.ec2.internal,UID:ip-10-0-130-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-189.ec2.internal,},FirstTimestamp:2026-05-27 23:33:29.827559432 +0000 UTC m=+0.485421897,LastTimestamp:2026-05-27 23:33:29.827559432 +0000 UTC m=+0.485421897,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-189.ec2.internal,}"
May 27 23:33:29.841558 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.841540    2562 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 27 23:33:29.842286 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.842268    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 27 23:33:29.842286 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.842283    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 27 23:33:29.842413 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.842352    2562 volume_manager.go:295] "The desired_state_of_world populator starts"
May 27 23:33:29.842413 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.842362    2562 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 27 23:33:29.842498 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.842452    2562 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 27 23:33:29.842577 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.842500    2562 reconstruct.go:97] "Volume reconstruction finished"
May 27 23:33:29.842577 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.842509    2562 reconciler.go:26] "Reconciler: start to sync state"
May 27 23:33:29.842937 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.842916    2562 factory.go:55] Registering systemd factory
May 27 23:33:29.842937 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.842939    2562 factory.go:223] Registration of the systemd container factory successfully
May 27 23:33:29.843162 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.843148    2562 factory.go:153] Registering CRI-O factory
May 27 23:33:29.843162 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.843165    2562 factory.go:223] Registration of the crio container factory successfully
May 27 23:33:29.843277 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.843209    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 27 23:33:29.843277 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.843227    2562 factory.go:103] Registering Raw factory
May 27 23:33:29.843277 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.843240    2562 manager.go:1196] Started watching for new ooms in manager
May 27 23:33:29.843418 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.843333    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:29.843717 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.843697    2562 manager.go:319] Starting recovery of all containers
May 27 23:33:29.845211 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.845189    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-189.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 27 23:33:29.845332 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.845309    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 27 23:33:29.854312 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.854228    2562 manager.go:324] Recovery completed
May 27 23:33:29.858300 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.858283    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:33:29.860143 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.860127    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:33:29.860205 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.860156    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:33:29.860205 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.860166    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
May 27 23:33:29.860845 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.860827    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 27 23:33:29.860943 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.860930    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 27 23:33:29.861019 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.861006    2562 state_mem.go:36] "Initialized new in-memory state store"
May 27 23:33:29.862822 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.862753    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-189.ec2.internal.18b3902f229e5930  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-189.ec2.internal,UID:ip-10-0-130-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-189.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-189.ec2.internal,},FirstTimestamp:2026-05-27 23:33:29.860143408 +0000 UTC m=+0.518005873,LastTimestamp:2026-05-27 23:33:29.860143408 +0000 UTC m=+0.518005873,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-189.ec2.internal,}"
May 27 23:33:29.865363 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.865348    2562 policy_none.go:49] "None policy: Start"
May 27 23:33:29.865429 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.865369    2562 memory_manager.go:186] "Starting memorymanager" policy="None"
May 27 23:33:29.865429 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.865383    2562 state_mem.go:35] "Initializing new in-memory state store"
May 27 23:33:29.873917 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.873851    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-189.ec2.internal.18b3902f229e9ed2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-189.ec2.internal,UID:ip-10-0-130-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-189.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-189.ec2.internal,},FirstTimestamp:2026-05-27 23:33:29.860161234 +0000 UTC m=+0.518023699,LastTimestamp:2026-05-27 23:33:29.860161234 +0000 UTC m=+0.518023699,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-189.ec2.internal,}"
May 27 23:33:29.883104 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.883080    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5h4lg"
May 27 23:33:29.883104 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.883040    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-189.ec2.internal.18b3902f229ec202  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-189.ec2.internal,UID:ip-10-0-130-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-189.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-189.ec2.internal,},FirstTimestamp:2026-05-27 23:33:29.860170242 +0000 UTC m=+0.518032706,LastTimestamp:2026-05-27 23:33:29.860170242 +0000 UTC m=+0.518032706,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-189.ec2.internal,}"
May 27 23:33:29.890503 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.890486    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5h4lg"
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.903948    2562 manager.go:341] "Starting Device Plugin manager"
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.903974    2562 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.903985    2562 server.go:85] "Starting device plugin registration server"
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.904217    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.904229    2562 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.904326    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.904407    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.904416    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.904974    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 27 23:33:29.919781 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.905010    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:29.938197 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.938164    2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 27 23:33:29.939359 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.939343    2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 27 23:33:29.939450 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.939364    2562 status_manager.go:230] "Starting to sync pod status with apiserver"
May 27 23:33:29.939450 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.939381    2562 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 27 23:33:29.939450 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.939387    2562 kubelet.go:2451] "Starting kubelet main sync loop"
May 27 23:33:29.939597 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:29.939461    2562 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 27 23:33:29.945260 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:29.945244    2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:33:30.004392 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.004368    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:33:30.006623 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.006602    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:33:30.006729 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.006636    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:33:30.006729 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.006651    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
May 27 23:33:30.006729 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.006680    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-189.ec2.internal"
May 27 23:33:30.014667 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.014651    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-189.ec2.internal"
May 27 23:33:30.014723 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.014674    2562 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-189.ec2.internal\": node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.040566 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.040524    2562 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"]
May 27 23:33:30.040682 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.040618    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:33:30.041486 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.041470    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:33:30.041592 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.041502    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:33:30.041592 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.041516    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
May 27 23:33:30.042823 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.042808    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:33:30.042906 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.042831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/657d08b43faadccb2b3da3444879ea9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"657d08b43faadccb2b3da3444879ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.042906 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.042860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/657d08b43faadccb2b3da3444879ea9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"657d08b43faadccb2b3da3444879ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.042995 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.042963    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.042995 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.042985    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:33:30.044626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.044043    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:33:30.044626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.044068    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:33:30.044626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.044082    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
May 27 23:33:30.044626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.044388    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:33:30.044626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.044406    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:33:30.044626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.044432    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
May 27 23:33:30.045623 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.045608    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.045684 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.045647    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:33:30.046363 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.046348    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:33:30.046418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.046374    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:33:30.046418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.046385    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
May 27 23:33:30.057135 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.057114    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.069337 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.069318    2562 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-189.ec2.internal\" not found" node="ip-10-0-130-189.ec2.internal"
May 27 23:33:30.073626 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.073611    2562 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-189.ec2.internal\" not found" node="ip-10-0-130-189.ec2.internal"
May 27 23:33:30.142991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.142965    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/657d08b43faadccb2b3da3444879ea9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"657d08b43faadccb2b3da3444879ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.143111 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.142994    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/657d08b43faadccb2b3da3444879ea9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"657d08b43faadccb2b3da3444879ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.143111 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.143023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33069df76ded22cb2f23f3545c885186-config\") pod \"kube-apiserver-proxy-ip-10-0-130-189.ec2.internal\" (UID: \"33069df76ded22cb2f23f3545c885186\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.143111 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.143074    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/657d08b43faadccb2b3da3444879ea9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"657d08b43faadccb2b3da3444879ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.143215 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.143127    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/657d08b43faadccb2b3da3444879ea9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"657d08b43faadccb2b3da3444879ea9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.157322 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.157301    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.243698 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.243673    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33069df76ded22cb2f23f3545c885186-config\") pod \"kube-apiserver-proxy-ip-10-0-130-189.ec2.internal\" (UID: \"33069df76ded22cb2f23f3545c885186\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.243769 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.243725    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33069df76ded22cb2f23f3545c885186-config\") pod \"kube-apiserver-proxy-ip-10-0-130-189.ec2.internal\" (UID: \"33069df76ded22cb2f23f3545c885186\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.257861 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.257804    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.358799 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.358753    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.371932 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.371908    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.376748 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.376731    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
May 27 23:33:30.459825 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.459787    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.560327 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.560271    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.660784 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.660755    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.740999 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.740966    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 27 23:33:30.741699 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.741138    2562 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 27 23:33:30.761451 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.761412    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.843302 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.843049    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 27 23:33:30.851238 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.851210    2562 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:33:30.853389 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.853370    2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 27 23:33:30.861961 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.861940    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:30.872188 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.872165    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z9ljv"
May 27 23:33:30.880697 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.880679    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z9ljv"
May 27 23:33:30.894219 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.894170    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-26 23:28:29 +0000 UTC" deadline="2027-12-19 06:56:33.835701796 +0000 UTC"
May 27 23:33:30.894219 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.894216    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13687h23m2.94149023s"
May 27 23:33:30.952651 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:30.952610    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33069df76ded22cb2f23f3545c885186.slice/crio-9913fb549f3b72c4dcb3e37bc0f568604ab72fdf57a77562c59435c04cc05141 WatchSource:0}: Error finding container 9913fb549f3b72c4dcb3e37bc0f568604ab72fdf57a77562c59435c04cc05141: Status 404 returned error can't find the container with id 9913fb549f3b72c4dcb3e37bc0f568604ab72fdf57a77562c59435c04cc05141
May 27 23:33:30.952859 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:30.952843    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657d08b43faadccb2b3da3444879ea9f.slice/crio-9b8378abf2ae72d28196741d081bc10d7dc76100c6331cebe05631e09142b2ba WatchSource:0}: Error finding container 9b8378abf2ae72d28196741d081bc10d7dc76100c6331cebe05631e09142b2ba: Status 404 returned error can't find the container with id 9b8378abf2ae72d28196741d081bc10d7dc76100c6331cebe05631e09142b2ba
May 27 23:33:30.956861 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:30.956842    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 23:33:30.962240 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:30.962217    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:31.059988 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.059957    2562 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:33:31.063013 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.062995    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:31.163614 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.163498    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:31.264046 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.264011    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:31.364867 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.364831    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
May 27 23:33:31.401089 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.401064    2562 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:33:31.443309 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.443225    2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
May 27 23:33:31.456839 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.456811    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 27 23:33:31.457961 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.457947    2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
May 27 23:33:31.466353 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.466332    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 27 23:33:31.816327 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.816293    2562 apiserver.go:52] "Watching apiserver"
May 27 23:33:31.824475 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.824445    2562 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 27 23:33:31.826127 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.826097    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jd94m","openshift-cluster-node-tuning-operator/tuned-cs54f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal","openshift-network-diagnostics/network-check-target-qfvtb","openshift-network-operator/iptables-alerter-c6tbg","openshift-ovn-kubernetes/ovnkube-node-khn4g","kube-system/konnectivity-agent-lzg9b","kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv","openshift-dns/node-resolver-dbcvm","openshift-image-registry/node-ca-j7qvm","openshift-multus/multus-additional-cni-plugins-cgd7k","openshift-multus/multus-s5s5h"]
May 27 23:33:31.828815 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.828794    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:31.831287 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.831267    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 27 23:33:31.831379 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.831271    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ftztt\""
May 27 23:33:31.831379 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.831356    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 27 23:33:31.833272 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.832948    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.835069 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.835040    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:31.835172 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.835119    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:31.835172 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.835144    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.835702 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.835519    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b2q82\""
May 27 23:33:31.835702 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.835583    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 27 23:33:31.835702 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.835618    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 27 23:33:31.837241 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.837218    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 27 23:33:31.838254 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.838235    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 27 23:33:31.838349 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.838276    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2z4kc\""
May 27 23:33:31.838349 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.838285    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 27 23:33:31.839583 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.839468    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.842483 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.840091    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:31.842483 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.840182    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:31.842483 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.842154    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 27 23:33:31.843746 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.842867    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5bhl7\""
May 27 23:33:31.843746 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.843100    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 27 23:33:31.843746 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.843373    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 27 23:33:31.843746 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.843519    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.844603 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.844225    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 27 23:33:31.844603 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.844389    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 27 23:33:31.844603 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.844428    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 27 23:33:31.847073 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.847053    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 27 23:33:31.847167 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.847118    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 27 23:33:31.847374 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.847238    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 27 23:33:31.847374 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.847344    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vtwgm\""
May 27 23:33:31.848551 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.848522    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:31.848937 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.848609    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:31.850978 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.850960    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 27 23:33:31.851066 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.851027    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 27 23:33:31.851066 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.850961    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 27 23:33:31.851187 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.851118    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:31.851187 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.850963    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8c6dc\""
May 27 23:33:31.851723 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.851707    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 27 23:33:31.851966 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.851834    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kstcf\""
May 27 23:33:31.852164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.852141    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 27 23:33:31.853109 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e2612fe-61ea-401d-a0b6-b7605e4999ec-ovn-node-metrics-cert\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.853207 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853123    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-lib-modules\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.853207 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853190    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7ww\" (UniqueName: \"kubernetes.io/projected/1d468bad-a43d-40a3-9fdc-9a800365e54c-kube-api-access-8m7ww\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.853317 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f493421e-4338-455e-b688-34046f4c4d05-agent-certs\") pod \"konnectivity-agent-lzg9b\" (UID: \"f493421e-4338-455e-b688-34046f4c4d05\") " pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:31.853317 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853257    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9sgs\" (UniqueName: \"kubernetes.io/projected/c4abd5dc-33fe-4e84-a433-06f2916dab75-kube-api-access-j9sgs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:31.853417 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-sysctl-d\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.853872 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853718    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z55wv\""
May 27 23:33:31.853872 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853752    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-sys\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.853872 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853470    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 27 23:33:31.853872 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853508    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 27 23:33:31.854077 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853640    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.854077 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853682    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 27 23:33:31.854172 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853697    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 27 23:33:31.854296 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.853586    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 27 23:33:31.854440 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-slash\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.854561 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-run-systemd\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.854561 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.854694 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e2612fe-61ea-401d-a0b6-b7605e4999ec-env-overrides\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.854694 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e2612fe-61ea-401d-a0b6-b7605e4999ec-ovnkube-script-lib\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.854694 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854644    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-socket-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.854835 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-device-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.854835 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.854835 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:31.854835 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-modprobe-d\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855023 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-sysconfig\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855023 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-var-lib-kubelet\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855023 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:31.855023 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.854991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-etc-openvswitch\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855023 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-cni-netd\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855272 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e2612fe-61ea-401d-a0b6-b7605e4999ec-ovnkube-config\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855272 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.855272 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d468bad-a43d-40a3-9fdc-9a800365e54c-tmp\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855272 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7c1c6064-ffa6-4522-b6b6-2c6a275fffea-iptables-alerter-script\") pod \"iptables-alerter-c6tbg\" (UID: \"7c1c6064-ffa6-4522-b6b6-2c6a275fffea\") " pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.855272 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c1c6064-ffa6-4522-b6b6-2c6a275fffea-host-slash\") pod \"iptables-alerter-c6tbg\" (UID: \"7c1c6064-ffa6-4522-b6b6-2c6a275fffea\") " pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.855272 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855205    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-run-ovn\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855272 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855237    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-log-socket\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855272 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-sys-fs\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.855604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f493421e-4338-455e-b688-34046f4c4d05-konnectivity-ca\") pod \"konnectivity-agent-lzg9b\" (UID: \"f493421e-4338-455e-b688-34046f4c4d05\") " pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:31.855604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-systemd\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855469    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-run-openvswitch\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-registration-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.855604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855549    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5snql\" (UniqueName: \"kubernetes.io/projected/bcac5cd9-193c-4f83-8716-aecaf1048b49-kube-api-access-5snql\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.855604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-kubernetes\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-sysctl-conf\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-run\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-host\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-run-netns\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-node-log\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qk7\" (UniqueName: \"kubernetes.io/projected/9e2612fe-61ea-401d-a0b6-b7605e4999ec-kube-api-access-t9qk7\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-tuned\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.855911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmgn\" (UniqueName: \"kubernetes.io/projected/7c1c6064-ffa6-4522-b6b6-2c6a275fffea-kube-api-access-9rmgn\") pod \"iptables-alerter-c6tbg\" (UID: \"7c1c6064-ffa6-4522-b6b6-2c6a275fffea\") " pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.855911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-kubelet\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.855911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-systemd-units\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.856354 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-var-lib-openvswitch\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.856354 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.855937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-cni-bin\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.856354 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.856031    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 27 23:33:31.856354 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.856314    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5nqwg\""
May 27 23:33:31.881245 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.881215    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-26 23:28:30 +0000 UTC" deadline="2028-01-07 14:09:26.392486053 +0000 UTC"
May 27 23:33:31.881351 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.881245    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14150h35m54.511245195s"
May 27 23:33:31.888899 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.888806    2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:33:31.945245 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.945171    2562 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 27 23:33:31.946039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.945980    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal" event={"ID":"33069df76ded22cb2f23f3545c885186","Type":"ContainerStarted","Data":"9913fb549f3b72c4dcb3e37bc0f568604ab72fdf57a77562c59435c04cc05141"}
May 27 23:33:31.947060 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.947026    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal" event={"ID":"657d08b43faadccb2b3da3444879ea9f","Type":"ContainerStarted","Data":"9b8378abf2ae72d28196741d081bc10d7dc76100c6331cebe05631e09142b2ba"}
May 27 23:33:31.956639 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956619    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-sys-fs\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.956738 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956649    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-kubernetes\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.956738 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956665    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-sysctl-conf\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.956738 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956681    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-run\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.956738 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956704    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-host\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-cnibin\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956755    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-sys-fs\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956756    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-kubernetes\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-daemon-config\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956798    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-tuned\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956819    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-host\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956830    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-run\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956826    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmgn\" (UniqueName: \"kubernetes.io/projected/7c1c6064-ffa6-4522-b6b6-2c6a275fffea-kube-api-access-9rmgn\") pod \"iptables-alerter-c6tbg\" (UID: \"7c1c6064-ffa6-4522-b6b6-2c6a275fffea\") " pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956889    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-var-lib-openvswitch\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956925    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j9sgs\" (UniqueName: \"kubernetes.io/projected/c4abd5dc-33fe-4e84-a433-06f2916dab75-kube-api-access-j9sgs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:31.956955 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-system-cni-dir\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956966    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-var-lib-openvswitch\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-socket-dir-parent\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.956983    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-sysctl-conf\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957029    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-run-k8s-cni-cncf-io\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957060    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-lib-modules\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f795878-59a3-488e-af3e-9df4e143020a-hosts-file\") pod \"node-resolver-dbcvm\" (UID: \"1f795878-59a3-488e-af3e-9df4e143020a\") " pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957113    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhb7z\" (UniqueName: \"kubernetes.io/projected/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-kube-api-access-mhb7z\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-hostroot\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957139    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957164    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-slash\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957191    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-run-systemd\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957202    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-lib-modules\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957216    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957241    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e2612fe-61ea-401d-a0b6-b7605e4999ec-ovnkube-script-lib\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957253    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-run-systemd\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk7t8\" (UniqueName: \"kubernetes.io/projected/1f795878-59a3-488e-af3e-9df4e143020a-kube-api-access-xk7t8\") pod \"node-resolver-dbcvm\" (UID: \"1f795878-59a3-488e-af3e-9df4e143020a\") " pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:31.957381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957246    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-slash\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-cni-dir\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957298    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b71eac2b-1dcb-4cfb-8008-3795f1abab85-cni-binary-copy\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957454    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-modprobe-d\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957487    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-var-lib-kubelet\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957557    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-var-lib-kubelet\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957562    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-etc-openvswitch\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957570    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-modprobe-d\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957580    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-cni-netd\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957595    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e2612fe-61ea-401d-a0b6-b7605e4999ec-ovnkube-config\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957602    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-etc-openvswitch\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-system-cni-dir\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-var-lib-cni-bin\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957641    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-cni-netd\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-var-lib-cni-multus\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.958164 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957677    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d468bad-a43d-40a3-9fdc-9a800365e54c-tmp\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957699    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7c1c6064-ffa6-4522-b6b6-2c6a275fffea-iptables-alerter-script\") pod \"iptables-alerter-c6tbg\" (UID: \"7c1c6064-ffa6-4522-b6b6-2c6a275fffea\") " pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957722    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-run-ovn\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957746    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957772    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f493421e-4338-455e-b688-34046f4c4d05-konnectivity-ca\") pod \"konnectivity-agent-lzg9b\" (UID: \"f493421e-4338-455e-b688-34046f4c4d05\") " pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957796    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a1534cf-d950-40c0-9a7e-18fc4fc88ee0-serviceca\") pod \"node-ca-j7qvm\" (UID: \"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0\") " pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957814    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-run-ovn\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-run-netns\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957880    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e2612fe-61ea-401d-a0b6-b7605e4999ec-ovnkube-script-lib\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-etc-kubernetes\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957874    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957926    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-systemd\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957959    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-registration-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.957989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-run-multus-certs\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958044    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-run-netns\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958060    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-registration-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.958931 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958068    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-node-log\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958109    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qk7\" (UniqueName: \"kubernetes.io/projected/9e2612fe-61ea-401d-a0b6-b7605e4999ec-kube-api-access-t9qk7\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958118    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-run-netns\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958135    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-kubelet\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958147    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e2612fe-61ea-401d-a0b6-b7605e4999ec-ovnkube-config\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958160    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-systemd-units\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958175    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-systemd\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958215    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-cni-bin\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958224    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-kubelet\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958247    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e2612fe-61ea-401d-a0b6-b7605e4999ec-ovn-node-metrics-cert\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958248    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7c1c6064-ffa6-4522-b6b6-2c6a275fffea-iptables-alerter-script\") pod \"iptables-alerter-c6tbg\" (UID: \"7c1c6064-ffa6-4522-b6b6-2c6a275fffea\") " pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958269    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-systemd-units\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958275    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7ww\" (UniqueName: \"kubernetes.io/projected/1d468bad-a43d-40a3-9fdc-9a800365e54c-kube-api-access-8m7ww\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958416    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-host-cni-bin\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958431    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f493421e-4338-455e-b688-34046f4c4d05-agent-certs\") pod \"konnectivity-agent-lzg9b\" (UID: \"f493421e-4338-455e-b688-34046f4c4d05\") " pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-cnibin\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958468    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f493421e-4338-455e-b688-34046f4c4d05-konnectivity-ca\") pod \"konnectivity-agent-lzg9b\" (UID: \"f493421e-4338-455e-b688-34046f4c4d05\") " pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:31.959807 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9tkf\" (UniqueName: \"kubernetes.io/projected/b71eac2b-1dcb-4cfb-8008-3795f1abab85-kube-api-access-f9tkf\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958515    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-node-log\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958646    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-sysctl-d\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958676    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-sys\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958729    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-sys\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958789    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-sysctl-d\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958822    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e2612fe-61ea-401d-a0b6-b7605e4999ec-env-overrides\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958851    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-socket-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958909    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-device-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958933    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.958964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-os-release\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959043    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959069    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-conf-dir\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959096    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-sysconfig\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959136    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959160    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.960662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959227    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a1534cf-d950-40c0-9a7e-18fc4fc88ee0-host\") pod \"node-ca-j7qvm\" (UID: \"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0\") " pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-os-release\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-var-lib-kubelet\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959321    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c1c6064-ffa6-4522-b6b6-2c6a275fffea-host-slash\") pod \"iptables-alerter-c6tbg\" (UID: \"7c1c6064-ffa6-4522-b6b6-2c6a275fffea\") " pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959324    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e2612fe-61ea-401d-a0b6-b7605e4999ec-env-overrides\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959361    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-log-socket\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959381    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c1c6064-ffa6-4522-b6b6-2c6a275fffea-host-slash\") pod \"iptables-alerter-c6tbg\" (UID: \"7c1c6064-ffa6-4522-b6b6-2c6a275fffea\") " pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959397    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-device-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4cr\" (UniqueName: \"kubernetes.io/projected/4a1534cf-d950-40c0-9a7e-18fc4fc88ee0-kube-api-access-sj4cr\") pod \"node-ca-j7qvm\" (UID: \"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0\") " pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959438    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-run-openvswitch\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959441    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.959459    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959498    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-log-socket\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959563    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-sysconfig\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.959567    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs podName:c4abd5dc-33fe-4e84-a433-06f2916dab75 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:32.459514951 +0000 UTC m=+3.117377430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs") pod "network-metrics-daemon-jd94m" (UID: "c4abd5dc-33fe-4e84-a433-06f2916dab75") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959584    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-socket-dir\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959602    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5snql\" (UniqueName: \"kubernetes.io/projected/bcac5cd9-193c-4f83-8716-aecaf1048b49-kube-api-access-5snql\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.961456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959625    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e2612fe-61ea-401d-a0b6-b7605e4999ec-run-openvswitch\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.962267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959630    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f795878-59a3-488e-af3e-9df4e143020a-tmp-dir\") pod \"node-resolver-dbcvm\" (UID: \"1f795878-59a3-488e-af3e-9df4e143020a\") " pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:31.962267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959668    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bcac5cd9-193c-4f83-8716-aecaf1048b49-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:31.962267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.959732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:31.962267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.960625    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d468bad-a43d-40a3-9fdc-9a800365e54c-tmp\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.962267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.960831    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d468bad-a43d-40a3-9fdc-9a800365e54c-etc-tuned\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.962267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.960948    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e2612fe-61ea-401d-a0b6-b7605e4999ec-ovn-node-metrics-cert\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.962267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.961170    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f493421e-4338-455e-b688-34046f4c4d05-agent-certs\") pod \"konnectivity-agent-lzg9b\" (UID: \"f493421e-4338-455e-b688-34046f4c4d05\") " pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:31.965859 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.965713    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmgn\" (UniqueName: \"kubernetes.io/projected/7c1c6064-ffa6-4522-b6b6-2c6a275fffea-kube-api-access-9rmgn\") pod \"iptables-alerter-c6tbg\" (UID: \"7c1c6064-ffa6-4522-b6b6-2c6a275fffea\") " pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:31.965859 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.965797    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9sgs\" (UniqueName: \"kubernetes.io/projected/c4abd5dc-33fe-4e84-a433-06f2916dab75-kube-api-access-j9sgs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:31.966006 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.965923    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:33:31.966006 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.965943    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:33:31.966006 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.965955    2562 projected.go:194] Error preparing data for projected volume kube-api-access-kkjqg for pod openshift-network-diagnostics/network-check-target-qfvtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:31.966006 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:31.966004    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg podName:c9d94568-264b-4508-9f30-a7bc58643904 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:32.465988551 +0000 UTC m=+3.123851022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kkjqg" (UniqueName: "kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg") pod "network-check-target-qfvtb" (UID: "c9d94568-264b-4508-9f30-a7bc58643904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:31.966523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.966482    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7ww\" (UniqueName: \"kubernetes.io/projected/1d468bad-a43d-40a3-9fdc-9a800365e54c-kube-api-access-8m7ww\") pod \"tuned-cs54f\" (UID: \"1d468bad-a43d-40a3-9fdc-9a800365e54c\") " pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:31.966644 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.966614    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qk7\" (UniqueName: \"kubernetes.io/projected/9e2612fe-61ea-401d-a0b6-b7605e4999ec-kube-api-access-t9qk7\") pod \"ovnkube-node-khn4g\" (UID: \"9e2612fe-61ea-401d-a0b6-b7605e4999ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:31.967695 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:31.967673    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5snql\" (UniqueName: \"kubernetes.io/projected/bcac5cd9-193c-4f83-8716-aecaf1048b49-kube-api-access-5snql\") pod \"aws-ebs-csi-driver-node-jjqtv\" (UID: \"bcac5cd9-193c-4f83-8716-aecaf1048b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:32.060547 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060496    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.060720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060560    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-run-multus-certs\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.060720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060582    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-cnibin\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.060720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060598    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tkf\" (UniqueName: \"kubernetes.io/projected/b71eac2b-1dcb-4cfb-8008-3795f1abab85-kube-api-access-f9tkf\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.060720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060622    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-os-release\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.060720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060645    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.060720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060662    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-run-multus-certs\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.060720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060688    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.060720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060689    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-cnibin\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060730    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-os-release\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060859    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-conf-dir\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060888    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a1534cf-d950-40c0-9a7e-18fc4fc88ee0-host\") pod \"node-ca-j7qvm\" (UID: \"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0\") " pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060904    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-os-release\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060922    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-var-lib-kubelet\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060947    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-conf-dir\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060958    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a1534cf-d950-40c0-9a7e-18fc4fc88ee0-host\") pod \"node-ca-j7qvm\" (UID: \"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0\") " pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.060949    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4cr\" (UniqueName: \"kubernetes.io/projected/4a1534cf-d950-40c0-9a7e-18fc4fc88ee0-kube-api-access-sj4cr\") pod \"node-ca-j7qvm\" (UID: \"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0\") " pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061000    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f795878-59a3-488e-af3e-9df4e143020a-tmp-dir\") pod \"node-resolver-dbcvm\" (UID: \"1f795878-59a3-488e-af3e-9df4e143020a\") " pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061023    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-var-lib-kubelet\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061000    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-os-release\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061024    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061066    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-cnibin\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.061092 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061092    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-daemon-config\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061120    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-cnibin\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061135    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061147    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-system-cni-dir\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061119    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-system-cni-dir\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061181    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-socket-dir-parent\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061199    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-run-k8s-cni-cncf-io\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061214    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f795878-59a3-488e-af3e-9df4e143020a-hosts-file\") pod \"node-resolver-dbcvm\" (UID: \"1f795878-59a3-488e-af3e-9df4e143020a\") " pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061233    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mhb7z\" (UniqueName: \"kubernetes.io/projected/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-kube-api-access-mhb7z\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061240    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-socket-dir-parent\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061252    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f795878-59a3-488e-af3e-9df4e143020a-tmp-dir\") pod \"node-resolver-dbcvm\" (UID: \"1f795878-59a3-488e-af3e-9df4e143020a\") " pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061256    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-hostroot\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061282    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-run-k8s-cni-cncf-io\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061284    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-hostroot\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061303    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f795878-59a3-488e-af3e-9df4e143020a-hosts-file\") pod \"node-resolver-dbcvm\" (UID: \"1f795878-59a3-488e-af3e-9df4e143020a\") " pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061308    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xk7t8\" (UniqueName: \"kubernetes.io/projected/1f795878-59a3-488e-af3e-9df4e143020a-kube-api-access-xk7t8\") pod \"node-resolver-dbcvm\" (UID: \"1f795878-59a3-488e-af3e-9df4e143020a\") " pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061334    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-cni-dir\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.061745 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061357    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b71eac2b-1dcb-4cfb-8008-3795f1abab85-cni-binary-copy\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061382    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061413    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-system-cni-dir\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061426    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-cni-dir\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061438    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-var-lib-cni-bin\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061486    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-var-lib-cni-multus\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061520    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a1534cf-d950-40c0-9a7e-18fc4fc88ee0-serviceca\") pod \"node-ca-j7qvm\" (UID: \"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0\") " pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061570    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-run-netns\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061572    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-var-lib-cni-bin\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061590    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-etc-kubernetes\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061630    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-system-cni-dir\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061636    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-etc-kubernetes\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061663    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-run-netns\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061666    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b71eac2b-1dcb-4cfb-8008-3795f1abab85-multus-daemon-config\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061688    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b71eac2b-1dcb-4cfb-8008-3795f1abab85-host-var-lib-cni-multus\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061849    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b71eac2b-1dcb-4cfb-8008-3795f1abab85-cni-binary-copy\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.061951    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.062481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.062006    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a1534cf-d950-40c0-9a7e-18fc4fc88ee0-serviceca\") pod \"node-ca-j7qvm\" (UID: \"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0\") " pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:32.063227 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.062573    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.069038 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.068984    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhb7z\" (UniqueName: \"kubernetes.io/projected/7ebdbd61-602f-4e9b-95a1-ed43b4d13158-kube-api-access-mhb7z\") pod \"multus-additional-cni-plugins-cgd7k\" (UID: \"7ebdbd61-602f-4e9b-95a1-ed43b4d13158\") " pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.069405 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.069384    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4cr\" (UniqueName: \"kubernetes.io/projected/4a1534cf-d950-40c0-9a7e-18fc4fc88ee0-kube-api-access-sj4cr\") pod \"node-ca-j7qvm\" (UID: \"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0\") " pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:32.069930 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.069906    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tkf\" (UniqueName: \"kubernetes.io/projected/b71eac2b-1dcb-4cfb-8008-3795f1abab85-kube-api-access-f9tkf\") pod \"multus-s5s5h\" (UID: \"b71eac2b-1dcb-4cfb-8008-3795f1abab85\") " pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.070089 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.070066    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk7t8\" (UniqueName: \"kubernetes.io/projected/1f795878-59a3-488e-af3e-9df4e143020a-kube-api-access-xk7t8\") pod \"node-resolver-dbcvm\" (UID: \"1f795878-59a3-488e-af3e-9df4e143020a\") " pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:32.141036 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.140998    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:32.149045 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.149016    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cs54f"
May 27 23:33:32.160778 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.160755    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c6tbg"
May 27 23:33:32.166558 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.166523    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:32.174120 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.174100    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv"
May 27 23:33:32.181561 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.181522    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dbcvm"
May 27 23:33:32.189242 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.189216    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j7qvm"
May 27 23:33:32.197778 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.197760    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgd7k"
May 27 23:33:32.203361 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.203342    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5s5h"
May 27 23:33:32.464270 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.464192    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:32.464427 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:32.464362    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:32.464482 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:32.464439    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs podName:c4abd5dc-33fe-4e84-a433-06f2916dab75 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:33.464418249 +0000 UTC m=+4.122280707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs") pod "network-metrics-daemon-jd94m" (UID: "c4abd5dc-33fe-4e84-a433-06f2916dab75") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:32.533516 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:32.533475    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1c6064_ffa6_4522_b6b6_2c6a275fffea.slice/crio-2be07af096b237103d3a5f8698e84f342b321b242530f169ee690939a31064a4 WatchSource:0}: Error finding container 2be07af096b237103d3a5f8698e84f342b321b242530f169ee690939a31064a4: Status 404 returned error can't find the container with id 2be07af096b237103d3a5f8698e84f342b321b242530f169ee690939a31064a4
May 27 23:33:32.534746 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:32.534689    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1534cf_d950_40c0_9a7e_18fc4fc88ee0.slice/crio-d2476ef84f6d8533b9c5f1f4f887a94fbf37065edc462b660045c006ccc1fe85 WatchSource:0}: Error finding container d2476ef84f6d8533b9c5f1f4f887a94fbf37065edc462b660045c006ccc1fe85: Status 404 returned error can't find the container with id d2476ef84f6d8533b9c5f1f4f887a94fbf37065edc462b660045c006ccc1fe85
May 27 23:33:32.535440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:32.535418    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcac5cd9_193c_4f83_8716_aecaf1048b49.slice/crio-f12bbc6058ebc7c1726bd4a61c6f825181a7ef3eae413b874553b874b88e2836 WatchSource:0}: Error finding container f12bbc6058ebc7c1726bd4a61c6f825181a7ef3eae413b874553b874b88e2836: Status 404 returned error can't find the container with id f12bbc6058ebc7c1726bd4a61c6f825181a7ef3eae413b874553b874b88e2836
May 27 23:33:32.540251 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:32.540225    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2612fe_61ea_401d_a0b6_b7605e4999ec.slice/crio-2006dd200295153035099a9aa288497f7f68cdebb2f8a36e759f7c39a98e696f WatchSource:0}: Error finding container 2006dd200295153035099a9aa288497f7f68cdebb2f8a36e759f7c39a98e696f: Status 404 returned error can't find the container with id 2006dd200295153035099a9aa288497f7f68cdebb2f8a36e759f7c39a98e696f
May 27 23:33:32.562440 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:32.562388    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71eac2b_1dcb_4cfb_8008_3795f1abab85.slice/crio-473313c63949e07ecab2a9a6587fa2e9905c3214f3103771dcfee6c7141bd2ce WatchSource:0}: Error finding container 473313c63949e07ecab2a9a6587fa2e9905c3214f3103771dcfee6c7141bd2ce: Status 404 returned error can't find the container with id 473313c63949e07ecab2a9a6587fa2e9905c3214f3103771dcfee6c7141bd2ce
May 27 23:33:32.564552 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.564478    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:32.564708 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:32.564685    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:33:32.564708 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:32.564709    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:33:32.564921 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:32.564722    2562 projected.go:194] Error preparing data for projected volume kube-api-access-kkjqg for pod openshift-network-diagnostics/network-check-target-qfvtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:32.564921 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:32.564773    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg podName:c9d94568-264b-4508-9f30-a7bc58643904 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:33.564754075 +0000 UTC m=+4.222616530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkjqg" (UniqueName: "kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg") pod "network-check-target-qfvtb" (UID: "c9d94568-264b-4508-9f30-a7bc58643904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:32.565335 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:32.565287    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f795878_59a3_488e_af3e_9df4e143020a.slice/crio-de22e92f52a3b055ee33f1a8352b3895e57c66f70b0f147a92f05b63c9836873 WatchSource:0}: Error finding container de22e92f52a3b055ee33f1a8352b3895e57c66f70b0f147a92f05b63c9836873: Status 404 returned error can't find the container with id de22e92f52a3b055ee33f1a8352b3895e57c66f70b0f147a92f05b63c9836873
May 27 23:33:32.565956 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:32.565929    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf493421e_4338_455e_b688_34046f4c4d05.slice/crio-e8e1edacf3f8fe129d7bf7c5e53682be65539b13b18d9fd00bd3d870c853df6a WatchSource:0}: Error finding container e8e1edacf3f8fe129d7bf7c5e53682be65539b13b18d9fd00bd3d870c853df6a: Status 404 returned error can't find the container with id e8e1edacf3f8fe129d7bf7c5e53682be65539b13b18d9fd00bd3d870c853df6a
May 27 23:33:32.566814 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:32.566793    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d468bad_a43d_40a3_9fdc_9a800365e54c.slice/crio-299722ab447b3b720a434d9ef97c211eea6c48db557a08d3d1e1682a8e7a24d6 WatchSource:0}: Error finding container 299722ab447b3b720a434d9ef97c211eea6c48db557a08d3d1e1682a8e7a24d6: Status 404 returned error can't find the container with id 299722ab447b3b720a434d9ef97c211eea6c48db557a08d3d1e1682a8e7a24d6
May 27 23:33:32.567824 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:33:32.567796    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebdbd61_602f_4e9b_95a1_ed43b4d13158.slice/crio-ce8b1f3a9e659a7e4c92b2e81cd6dd9e97d9d0e68ae8a37db52727e4a3e7d376 WatchSource:0}: Error finding container ce8b1f3a9e659a7e4c92b2e81cd6dd9e97d9d0e68ae8a37db52727e4a3e7d376: Status 404 returned error can't find the container with id ce8b1f3a9e659a7e4c92b2e81cd6dd9e97d9d0e68ae8a37db52727e4a3e7d376
May 27 23:33:32.881729 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.881659    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-26 23:28:30 +0000 UTC" deadline="2028-02-06 09:59:45.941017622 +0000 UTC"
May 27 23:33:32.881729 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.881696    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14866h26m13.059324897s"
May 27 23:33:32.940262 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.940231    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:32.940448 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:32.940359    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:32.951811 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.951773    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j7qvm" event={"ID":"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0","Type":"ContainerStarted","Data":"d2476ef84f6d8533b9c5f1f4f887a94fbf37065edc462b660045c006ccc1fe85"}
May 27 23:33:32.953852 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.953819    2562 generic.go:358] "Generic (PLEG): container finished" podID="657d08b43faadccb2b3da3444879ea9f" containerID="286a23130cca1a50db7c052fae69d0e02574f1f9a003401a9186a653bc7a55c0" exitCode=0
May 27 23:33:32.953972 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.953895    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal" event={"ID":"657d08b43faadccb2b3da3444879ea9f","Type":"ContainerDied","Data":"286a23130cca1a50db7c052fae69d0e02574f1f9a003401a9186a653bc7a55c0"}
May 27 23:33:32.958926 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.958859    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgd7k" event={"ID":"7ebdbd61-602f-4e9b-95a1-ed43b4d13158","Type":"ContainerStarted","Data":"ce8b1f3a9e659a7e4c92b2e81cd6dd9e97d9d0e68ae8a37db52727e4a3e7d376"}
May 27 23:33:32.963904 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.963848    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cs54f" event={"ID":"1d468bad-a43d-40a3-9fdc-9a800365e54c","Type":"ContainerStarted","Data":"299722ab447b3b720a434d9ef97c211eea6c48db557a08d3d1e1682a8e7a24d6"}
May 27 23:33:32.965334 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.965275    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lzg9b" event={"ID":"f493421e-4338-455e-b688-34046f4c4d05","Type":"ContainerStarted","Data":"e8e1edacf3f8fe129d7bf7c5e53682be65539b13b18d9fd00bd3d870c853df6a"}
May 27 23:33:32.967991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.967588    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv" event={"ID":"bcac5cd9-193c-4f83-8716-aecaf1048b49","Type":"ContainerStarted","Data":"f12bbc6058ebc7c1726bd4a61c6f825181a7ef3eae413b874553b874b88e2836"}
May 27 23:33:32.969162 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.969135    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c6tbg" event={"ID":"7c1c6064-ffa6-4522-b6b6-2c6a275fffea","Type":"ContainerStarted","Data":"2be07af096b237103d3a5f8698e84f342b321b242530f169ee690939a31064a4"}
May 27 23:33:32.972415 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.972391    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal" event={"ID":"33069df76ded22cb2f23f3545c885186","Type":"ContainerStarted","Data":"bde0ce155e409cdfe274846b5f82df8c3a956e415885304c2058bb1cf142c44d"}
May 27 23:33:32.978657 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.978595    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dbcvm" event={"ID":"1f795878-59a3-488e-af3e-9df4e143020a","Type":"ContainerStarted","Data":"de22e92f52a3b055ee33f1a8352b3895e57c66f70b0f147a92f05b63c9836873"}
May 27 23:33:32.982035 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.982013    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5s5h" event={"ID":"b71eac2b-1dcb-4cfb-8008-3795f1abab85","Type":"ContainerStarted","Data":"473313c63949e07ecab2a9a6587fa2e9905c3214f3103771dcfee6c7141bd2ce"}
May 27 23:33:32.983271 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:32.983235    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerStarted","Data":"2006dd200295153035099a9aa288497f7f68cdebb2f8a36e759f7c39a98e696f"}
May 27 23:33:33.471102 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:33.471015    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:33.471260 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:33.471186    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:33.471260 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:33.471249    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs podName:c4abd5dc-33fe-4e84-a433-06f2916dab75 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:35.471229958 +0000 UTC m=+6.129092410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs") pod "network-metrics-daemon-jd94m" (UID: "c4abd5dc-33fe-4e84-a433-06f2916dab75") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:33.572155 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:33.572117    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:33.572324 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:33.572310    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:33:33.572379 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:33.572331    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:33:33.572379 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:33.572344    2562 projected.go:194] Error preparing data for projected volume kube-api-access-kkjqg for pod openshift-network-diagnostics/network-check-target-qfvtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:33.572495 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:33.572402    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg podName:c9d94568-264b-4508-9f30-a7bc58643904 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:35.572384325 +0000 UTC m=+6.230246789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkjqg" (UniqueName: "kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg") pod "network-check-target-qfvtb" (UID: "c9d94568-264b-4508-9f30-a7bc58643904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:33.941191 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:33.940690    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:33.941191 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:33.940835    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:33.999648 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:33.999580    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal" event={"ID":"657d08b43faadccb2b3da3444879ea9f","Type":"ContainerStarted","Data":"a52698becd2520cdf582e263e6c435c56818d366b4c39c81501f1e3e3eeefa8c"}
May 27 23:33:34.014959 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:34.014637    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal" podStartSLOduration=3.014617471 podStartE2EDuration="3.014617471s" podCreationTimestamp="2026-05-27 23:33:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:33:32.987687576 +0000 UTC m=+3.645550051" watchObservedRunningTime="2026-05-27 23:33:34.014617471 +0000 UTC m=+4.672479946"
May 27 23:33:34.940134 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:34.940103    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:34.940315 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:34.940238    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:35.489147 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:35.489106    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:35.489671 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:35.489297    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:35.489671 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:35.489362    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs podName:c4abd5dc-33fe-4e84-a433-06f2916dab75 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:39.489343452 +0000 UTC m=+10.147205922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs") pod "network-metrics-daemon-jd94m" (UID: "c4abd5dc-33fe-4e84-a433-06f2916dab75") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:35.590674 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:35.590571    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:35.590840 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:35.590757    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:33:35.590840 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:35.590778    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:33:35.590840 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:35.590791    2562 projected.go:194] Error preparing data for projected volume kube-api-access-kkjqg for pod openshift-network-diagnostics/network-check-target-qfvtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:35.591002 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:35.590856    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg podName:c9d94568-264b-4508-9f30-a7bc58643904 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:39.590838766 +0000 UTC m=+10.248701227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkjqg" (UniqueName: "kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg") pod "network-check-target-qfvtb" (UID: "c9d94568-264b-4508-9f30-a7bc58643904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:35.939737 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:35.939662    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:35.939882 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:35.939803    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:36.939866 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:36.939834    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:36.940309 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:36.939960    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:37.940134 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:37.940100    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:37.940604 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:37.940248    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:38.940381 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:38.940348    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:38.940870 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:38.940487    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:39.522378 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:39.522335    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:39.522596 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:39.522543    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:39.522668 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:39.522621    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs podName:c4abd5dc-33fe-4e84-a433-06f2916dab75 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:47.522599485 +0000 UTC m=+18.180461942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs") pod "network-metrics-daemon-jd94m" (UID: "c4abd5dc-33fe-4e84-a433-06f2916dab75") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:39.623372 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:39.623330    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:39.623563 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:39.623500    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:33:39.623563 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:39.623520    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:33:39.623563 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:39.623547    2562 projected.go:194] Error preparing data for projected volume kube-api-access-kkjqg for pod openshift-network-diagnostics/network-check-target-qfvtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:39.623786 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:39.623613    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg podName:c9d94568-264b-4508-9f30-a7bc58643904 nodeName:}" failed. No retries permitted until 2026-05-27 23:33:47.62359417 +0000 UTC m=+18.281456625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkjqg" (UniqueName: "kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg") pod "network-check-target-qfvtb" (UID: "c9d94568-264b-4508-9f30-a7bc58643904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:39.941023 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:39.940941    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:39.941459 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:39.941065    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:40.939811 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:40.939778    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:40.939983 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:40.939923    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:41.940223 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:41.940035    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:41.940648 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:41.940316    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:42.939818 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:42.939783    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:42.939990 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:42.939907    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:43.940218 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:43.940182    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:43.940784 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:43.940324    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:44.939864 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:44.939832    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:44.940019 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:44.939954    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:45.940409 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:45.940377    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:45.940890 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:45.940522    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:46.939857 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:46.939760    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:46.940043 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:46.939892    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:47.575686 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:47.575645    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:47.576133 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:47.575816    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:47.576133 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:47.575892    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs podName:c4abd5dc-33fe-4e84-a433-06f2916dab75 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:03.575869631 +0000 UTC m=+34.233732099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs") pod "network-metrics-daemon-jd94m" (UID: "c4abd5dc-33fe-4e84-a433-06f2916dab75") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:33:47.676644 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:47.676608    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:47.676797 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:47.676762    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:33:47.676797 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:47.676776    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:33:47.676797 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:47.676786    2562 projected.go:194] Error preparing data for projected volume kube-api-access-kkjqg for pod openshift-network-diagnostics/network-check-target-qfvtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:47.676927 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:47.676837    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg podName:c9d94568-264b-4508-9f30-a7bc58643904 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:03.676824435 +0000 UTC m=+34.334686898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkjqg" (UniqueName: "kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg") pod "network-check-target-qfvtb" (UID: "c9d94568-264b-4508-9f30-a7bc58643904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:33:47.939690 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:47.939654    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:47.939871 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:47.939782    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:48.939660 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:48.939625    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:48.940078 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:48.939762    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:49.940936 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:49.940908    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:49.941367 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:49.941024    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:50.939948 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:50.939813    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:50.940050 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:50.940029    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:51.036202 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.035888    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dbcvm" event={"ID":"1f795878-59a3-488e-af3e-9df4e143020a","Type":"ContainerStarted","Data":"ed40529a31062abc54b260a3b66227027a6544bb50c56163987d95cf826a4340"}
May 27 23:33:51.037788 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.037756    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5s5h" event={"ID":"b71eac2b-1dcb-4cfb-8008-3795f1abab85","Type":"ContainerStarted","Data":"2eae4ad49e698900c20cdaee857b51cedc9622fe3739cde5b2192559d0b9650c"}
May 27 23:33:51.040379 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.040347    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:33:51.040725 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.040699    2562 generic.go:358] "Generic (PLEG): container finished" podID="9e2612fe-61ea-401d-a0b6-b7605e4999ec" containerID="5bd57d465ed3f9c0e1bd3d6493dc08243b8cfa38f0f086c5d25ad989f08ac33a" exitCode=1
May 27 23:33:51.040825 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.040762    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerStarted","Data":"55192d8608d7716b9c9129ab1f739a0653e5f77e12aab63cf3757d057ec6e360"}
May 27 23:33:51.040825 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.040793    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerStarted","Data":"1c04e7c46ae044a0d00928c36fee9320fc6082152f4e767eaa3b26a7b7749500"}
May 27 23:33:51.040825 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.040807    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerDied","Data":"5bd57d465ed3f9c0e1bd3d6493dc08243b8cfa38f0f086c5d25ad989f08ac33a"}
May 27 23:33:51.040825 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.040822    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerStarted","Data":"fb934a05cf2e82e2b1e82fa27091eb68044d1db6ba554aaee3883565bbb9bcb5"}
May 27 23:33:51.042338 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.042316    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j7qvm" event={"ID":"4a1534cf-d950-40c0-9a7e-18fc4fc88ee0","Type":"ContainerStarted","Data":"3a8d22bd1b7da20bc3e9a435b00f2da5970c3a1059b5d52666056027b2c6f1d9"}
May 27 23:33:51.044193 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.044170    2562 generic.go:358] "Generic (PLEG): container finished" podID="7ebdbd61-602f-4e9b-95a1-ed43b4d13158" containerID="46217cdd6416fa013330f1e6d664980582e3e6ccef1e33b9a6ec874627ccaeb1" exitCode=0
May 27 23:33:51.044270 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.044246    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgd7k" event={"ID":"7ebdbd61-602f-4e9b-95a1-ed43b4d13158","Type":"ContainerDied","Data":"46217cdd6416fa013330f1e6d664980582e3e6ccef1e33b9a6ec874627ccaeb1"}
May 27 23:33:51.045951 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.045715    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cs54f" event={"ID":"1d468bad-a43d-40a3-9fdc-9a800365e54c","Type":"ContainerStarted","Data":"9527a2aaaa82802d13ca2c36a6beaf3c77558dc756f2326cba6a468ef919659b"}
May 27 23:33:51.049240 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.049215    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lzg9b" event={"ID":"f493421e-4338-455e-b688-34046f4c4d05","Type":"ContainerStarted","Data":"0a163aa01641a509b0f09c71a207ddca13860be64a116c2d2b5e11aa0df0c3b7"}
May 27 23:33:51.051117 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.051093    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv" event={"ID":"bcac5cd9-193c-4f83-8716-aecaf1048b49","Type":"ContainerStarted","Data":"ef903f114a621bfd2c71f90155add798e47b453e5bb50ee4742c465f40f36ff6"}
May 27 23:33:51.052219 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.052178    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal" podStartSLOduration=20.052162392 podStartE2EDuration="20.052162392s" podCreationTimestamp="2026-05-27 23:33:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:33:34.023813435 +0000 UTC m=+4.681675910" watchObservedRunningTime="2026-05-27 23:33:51.052162392 +0000 UTC m=+21.710024867"
May 27 23:33:51.052861 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.052822    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-dbcvm" podStartSLOduration=3.205036854 podStartE2EDuration="21.052801821s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:33:32.566995675 +0000 UTC m=+3.224858127" lastFinishedPulling="2026-05-27 23:33:50.414760639 +0000 UTC m=+21.072623094" observedRunningTime="2026-05-27 23:33:51.051884629 +0000 UTC m=+21.709747104" watchObservedRunningTime="2026-05-27 23:33:51.052801821 +0000 UTC m=+21.710664288"
May 27 23:33:51.068595 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.068541    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lzg9b" podStartSLOduration=3.224449061 podStartE2EDuration="21.06850796s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:33:32.570895266 +0000 UTC m=+3.228757721" lastFinishedPulling="2026-05-27 23:33:50.414954158 +0000 UTC m=+21.072816620" observedRunningTime="2026-05-27 23:33:51.068061426 +0000 UTC m=+21.725923900" watchObservedRunningTime="2026-05-27 23:33:51.06850796 +0000 UTC m=+21.726370512"
May 27 23:33:51.084431 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.084320    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cs54f" podStartSLOduration=3.238681217 podStartE2EDuration="21.084303319s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:33:32.570921901 +0000 UTC m=+3.228784363" lastFinishedPulling="2026-05-27 23:33:50.416543995 +0000 UTC m=+21.074406465" observedRunningTime="2026-05-27 23:33:51.084235392 +0000 UTC m=+21.742097886" watchObservedRunningTime="2026-05-27 23:33:51.084303319 +0000 UTC m=+21.742165793"
May 27 23:33:51.101713 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.101661    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j7qvm" podStartSLOduration=11.225934642 podStartE2EDuration="21.101642169s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:33:32.537170239 +0000 UTC m=+3.195032695" lastFinishedPulling="2026-05-27 23:33:42.412877755 +0000 UTC m=+13.070740222" observedRunningTime="2026-05-27 23:33:51.100385872 +0000 UTC m=+21.758248346" watchObservedRunningTime="2026-05-27 23:33:51.101642169 +0000 UTC m=+21.759504644"
May 27 23:33:51.120678 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.120631    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-s5s5h" podStartSLOduration=3.262796135 podStartE2EDuration="21.120613384s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:33:32.565091188 +0000 UTC m=+3.222953643" lastFinishedPulling="2026-05-27 23:33:50.42290844 +0000 UTC m=+21.080770892" observedRunningTime="2026-05-27 23:33:51.120117993 +0000 UTC m=+21.777980470" watchObservedRunningTime="2026-05-27 23:33:51.120613384 +0000 UTC m=+21.778475857"
May 27 23:33:51.780677 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.780446    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 27 23:33:51.915328 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.915209    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-27T23:33:51.780670907Z","UUID":"df215b6e-d5c9-484d-8409-96645f3625c6","Handler":null,"Name":"","Endpoint":""}
May 27 23:33:51.918924 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.918898    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 27 23:33:51.918924 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.918931    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 27 23:33:51.939679 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:51.939652    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:51.939829 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:51.939773    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:52.054519 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:52.054481    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv" event={"ID":"bcac5cd9-193c-4f83-8716-aecaf1048b49","Type":"ContainerStarted","Data":"abc3db3fc0447a6490f04f238552381cf236718c1a2e92a955bb8c446839e976"}
May 27 23:33:52.056014 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:52.055971    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c6tbg" event={"ID":"7c1c6064-ffa6-4522-b6b6-2c6a275fffea","Type":"ContainerStarted","Data":"501ec2b3efb8298843f05178a98fdf39d4ef6a7a08d974711a9c75546781428b"}
May 27 23:33:52.058670 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:52.058649    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:33:52.059109 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:52.059061    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerStarted","Data":"fe3e9d621b54e166359ced66315a87bbccca718ed88a838fd888102e032626e1"}
May 27 23:33:52.059109 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:52.059094    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerStarted","Data":"68362fbfbfdf69402c7b98ee5e6ad5da6149049e8a5cdac7b55b602976d4564e"}
May 27 23:33:52.071138 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:52.071099    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c6tbg" podStartSLOduration=4.192673213 podStartE2EDuration="22.071086017s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:33:32.536888338 +0000 UTC m=+3.194750806" lastFinishedPulling="2026-05-27 23:33:50.415301152 +0000 UTC m=+21.073163610" observedRunningTime="2026-05-27 23:33:52.07092183 +0000 UTC m=+22.728784305" watchObservedRunningTime="2026-05-27 23:33:52.071086017 +0000 UTC m=+22.728948500"
May 27 23:33:52.187750 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:52.187671    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:52.188412 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:52.188392    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:52.939641 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:52.939610    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:52.939838 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:52.939746    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:53.065268 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:53.065216    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv" event={"ID":"bcac5cd9-193c-4f83-8716-aecaf1048b49","Type":"ContainerStarted","Data":"68d547586356ae0c3554d5d67dfe68fe9c92a5af46aa59710a509b994b00551b"}
May 27 23:33:53.084343 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:53.084258    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjqtv" podStartSLOduration=2.949784448 podStartE2EDuration="23.084244787s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:33:32.561281306 +0000 UTC m=+3.219143762" lastFinishedPulling="2026-05-27 23:33:52.695741635 +0000 UTC m=+23.353604101" observedRunningTime="2026-05-27 23:33:53.08362051 +0000 UTC m=+23.741482984" watchObservedRunningTime="2026-05-27 23:33:53.084244787 +0000 UTC m=+23.742107261"
May 27 23:33:53.940371 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:53.940334    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:53.940595 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:53.940504    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:54.069980 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:54.069943    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:33:54.070621 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:54.070273    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerStarted","Data":"2d0eb0a497172fae8b338a8583562e54b42762ba7de1b76fce2ed4f067be783f"}
May 27 23:33:54.070621 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:54.070332    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 27 23:33:54.939765 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:54.939732    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:54.939937 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:54.939844    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:55.939999 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:55.939832    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:55.940589 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:55.940084    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:56.076814 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.076784    2562 generic.go:358] "Generic (PLEG): container finished" podID="7ebdbd61-602f-4e9b-95a1-ed43b4d13158" containerID="939145561ee94f800f78df981c36c9f50a8d39fd506e87603b22ecc8f9eec9f9" exitCode=0
May 27 23:33:56.076999 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.076851    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgd7k" event={"ID":"7ebdbd61-602f-4e9b-95a1-ed43b4d13158","Type":"ContainerDied","Data":"939145561ee94f800f78df981c36c9f50a8d39fd506e87603b22ecc8f9eec9f9"}
May 27 23:33:56.080182 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.080165    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:33:56.080481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.080463    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerStarted","Data":"ddfc57d7e816f9917a64cd4c964e72fa570abfe5cc20b8690fee2fd394367754"}
May 27 23:33:56.080782 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.080765    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:56.080863 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.080788    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:56.080919 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.080905    2562 scope.go:117] "RemoveContainer" containerID="5bd57d465ed3f9c0e1bd3d6493dc08243b8cfa38f0f086c5d25ad989f08ac33a"
May 27 23:33:56.099662 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.097412    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:56.175523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.175490    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:56.175676 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.175644    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 27 23:33:56.176103 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.176085    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lzg9b"
May 27 23:33:56.939624 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:56.939585    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:56.939785 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:56.939712    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:57.086121 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:57.086094    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:33:57.086615 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:57.086484    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" event={"ID":"9e2612fe-61ea-401d-a0b6-b7605e4999ec","Type":"ContainerStarted","Data":"1a3b2937d2cd7b353f5341a4cfde7ad3b421d30583668583a50a0c9b5bb1c73c"}
May 27 23:33:57.086755 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:57.086736    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:57.101428 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:57.101406    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:33:57.116374 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:57.116324    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g" podStartSLOduration=9.195797083 podStartE2EDuration="27.116307582s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:33:32.561245375 +0000 UTC m=+3.219107840" lastFinishedPulling="2026-05-27 23:33:50.481755872 +0000 UTC m=+21.139618339" observedRunningTime="2026-05-27 23:33:57.115521547 +0000 UTC m=+27.773384044" watchObservedRunningTime="2026-05-27 23:33:57.116307582 +0000 UTC m=+27.774170062"
May 27 23:33:57.460199 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:57.460025    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qfvtb"]
May 27 23:33:57.460359 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:57.460342    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:57.460472 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:57.460449    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:57.460769 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:57.460740    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jd94m"]
May 27 23:33:57.460889 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:57.460853    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:57.460946 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:57.460922    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:33:58.090003 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:58.089970    2562 generic.go:358] "Generic (PLEG): container finished" podID="7ebdbd61-602f-4e9b-95a1-ed43b4d13158" containerID="dabe77560462d0c9ffc9457fe24e92d2539b9d9b9e1a603ce27436379a03dc60" exitCode=0
May 27 23:33:58.090365 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:58.090056    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgd7k" event={"ID":"7ebdbd61-602f-4e9b-95a1-ed43b4d13158","Type":"ContainerDied","Data":"dabe77560462d0c9ffc9457fe24e92d2539b9d9b9e1a603ce27436379a03dc60"}
May 27 23:33:58.940541 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:58.940497    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:33:58.940717 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:33:58.940503    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:33:58.940717 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:58.940620    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:33:58.940814 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:33:58.940727    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:34:00.096841 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:00.096810    2562 generic.go:358] "Generic (PLEG): container finished" podID="7ebdbd61-602f-4e9b-95a1-ed43b4d13158" containerID="498aba1f8be2f98d5e55ffea259f3ea6c4b52318b68e26f9ca52c38584ddf8d4" exitCode=0
May 27 23:34:00.097578 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:00.096879    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgd7k" event={"ID":"7ebdbd61-602f-4e9b-95a1-ed43b4d13158","Type":"ContainerDied","Data":"498aba1f8be2f98d5e55ffea259f3ea6c4b52318b68e26f9ca52c38584ddf8d4"}
May 27 23:34:00.939893 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:00.939861    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:34:00.940068 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:00.939861    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:34:00.940068 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:00.939970    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:34:00.940182 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:00.940088    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:34:02.940047 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:02.940010    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:34:02.940570 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:02.940011    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:34:02.940570 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:02.940137    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd94m" podUID="c4abd5dc-33fe-4e84-a433-06f2916dab75"
May 27 23:34:02.940570 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:02.940261    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfvtb" podUID="c9d94568-264b-4508-9f30-a7bc58643904"
May 27 23:34:03.598432 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:03.598398    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:34:03.598671 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:03.598584    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:34:03.598671 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:03.598666    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs podName:c4abd5dc-33fe-4e84-a433-06f2916dab75 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:35.598645415 +0000 UTC m=+66.256507871 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs") pod "network-metrics-daemon-jd94m" (UID: "c4abd5dc-33fe-4e84-a433-06f2916dab75") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:34:03.699589 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:03.699551    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:34:03.699768 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:03.699729    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:34:03.699768 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:03.699757    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:34:03.699862 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:03.699772    2562 projected.go:194] Error preparing data for projected volume kube-api-access-kkjqg for pod openshift-network-diagnostics/network-check-target-qfvtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:34:03.699862 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:03.699830    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg podName:c9d94568-264b-4508-9f30-a7bc58643904 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:35.699814729 +0000 UTC m=+66.357677181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkjqg" (UniqueName: "kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg") pod "network-check-target-qfvtb" (UID: "c9d94568-264b-4508-9f30-a7bc58643904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:34:04.122564 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.122519    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeReady"
May 27 23:34:04.123115 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.122697    2562 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 27 23:34:04.163961 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.163929    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6knhq"]
May 27 23:34:04.187851 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.187808    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hwcgw"]
May 27 23:34:04.188028 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.187989    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.190733 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.190519    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 27 23:34:04.190900 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.190624    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-szgzp\""
May 27 23:34:04.190988 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.190693    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 27 23:34:04.207961 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.207930    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwcgw"]
May 27 23:34:04.207961 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.207959    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:04.207961 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.207968    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6knhq"]
May 27 23:34:04.210607 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.210587    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 27 23:34:04.210719 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.210607    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 27 23:34:04.210785 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.210775    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lstvz\""
May 27 23:34:04.210843 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.210804    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 27 23:34:04.304782 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.304699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjtl7\" (UniqueName: \"kubernetes.io/projected/8accd220-388c-4d68-b5d9-f393b8474848-kube-api-access-rjtl7\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:04.304782 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.304744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-tmp-dir\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.304782 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.304782    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lsl\" (UniqueName: \"kubernetes.io/projected/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-kube-api-access-d5lsl\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.305059 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.304855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:04.305059 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.304957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-config-volume\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.305059 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.304983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.405781 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.405742    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-config-volume\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.406017 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.405787    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.406017 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.405831    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rjtl7\" (UniqueName: \"kubernetes.io/projected/8accd220-388c-4d68-b5d9-f393b8474848-kube-api-access-rjtl7\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:04.406017 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.405862    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-tmp-dir\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.406017 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:04.405997    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:34:04.406218 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:04.406075    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls podName:9c3a8dde-f0f1-4ebe-88ce-f56cd784f813 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:04.906052264 +0000 UTC m=+35.563914722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls") pod "dns-default-6knhq" (UID: "9c3a8dde-f0f1-4ebe-88ce-f56cd784f813") : secret "dns-default-metrics-tls" not found
May 27 23:34:04.406218 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.406119    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lsl\" (UniqueName: \"kubernetes.io/projected/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-kube-api-access-d5lsl\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.406218 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.406164    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:04.406860 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:04.406840    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:34:04.406946 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:04.406915    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert podName:8accd220-388c-4d68-b5d9-f393b8474848 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:04.906893325 +0000 UTC m=+35.564755795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert") pod "ingress-canary-hwcgw" (UID: "8accd220-388c-4d68-b5d9-f393b8474848") : secret "canary-serving-cert" not found
May 27 23:34:04.407165 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.407146    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-config-volume\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.407380 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.407363    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-tmp-dir\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.417208 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.417175    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjtl7\" (UniqueName: \"kubernetes.io/projected/8accd220-388c-4d68-b5d9-f393b8474848-kube-api-access-rjtl7\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:04.425183 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.425154    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lsl\" (UniqueName: \"kubernetes.io/projected/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-kube-api-access-d5lsl\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.911243 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.911206    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:04.911514 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.911273    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:04.911514 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:04.911363    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:34:04.911514 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:04.911422    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert podName:8accd220-388c-4d68-b5d9-f393b8474848 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:05.911404638 +0000 UTC m=+36.569267091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert") pod "ingress-canary-hwcgw" (UID: "8accd220-388c-4d68-b5d9-f393b8474848") : secret "canary-serving-cert" not found
May 27 23:34:04.911514 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:04.911363    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:34:04.911706 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:04.911549    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls podName:9c3a8dde-f0f1-4ebe-88ce-f56cd784f813 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:05.911517789 +0000 UTC m=+36.569380245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls") pod "dns-default-6knhq" (UID: "9c3a8dde-f0f1-4ebe-88ce-f56cd784f813") : secret "dns-default-metrics-tls" not found
May 27 23:34:04.939838 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.939795    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:34:04.940020 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.939795    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:34:04.943991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.943963    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 27 23:34:04.944142 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.944032    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 27 23:34:04.944142 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.943977    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptj7k\""
May 27 23:34:04.944142 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.944032    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 27 23:34:04.944316 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:04.943977    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8jlf4\""
May 27 23:34:05.917854 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:05.917820    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:05.918231 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:05.917879    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:05.918231 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:05.917969    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:34:05.918231 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:05.917982    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:34:05.918231 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:05.918032    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert podName:8accd220-388c-4d68-b5d9-f393b8474848 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:07.918016029 +0000 UTC m=+38.575878481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert") pod "ingress-canary-hwcgw" (UID: "8accd220-388c-4d68-b5d9-f393b8474848") : secret "canary-serving-cert" not found
May 27 23:34:05.918231 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:05.918046    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls podName:9c3a8dde-f0f1-4ebe-88ce-f56cd784f813 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:07.918039837 +0000 UTC m=+38.575902289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls") pod "dns-default-6knhq" (UID: "9c3a8dde-f0f1-4ebe-88ce-f56cd784f813") : secret "dns-default-metrics-tls" not found
May 27 23:34:07.110733 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:07.110701    2562 generic.go:358] "Generic (PLEG): container finished" podID="7ebdbd61-602f-4e9b-95a1-ed43b4d13158" containerID="5d11ecc95900f04b5553a029da65931df6b34063b30aa9422c0c6ff3261d1168" exitCode=0
May 27 23:34:07.111181 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:07.110754    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgd7k" event={"ID":"7ebdbd61-602f-4e9b-95a1-ed43b4d13158","Type":"ContainerDied","Data":"5d11ecc95900f04b5553a029da65931df6b34063b30aa9422c0c6ff3261d1168"}
May 27 23:34:07.933248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:07.933216    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:07.933415 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:07.933273    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:07.933415 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:07.933370    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:34:07.933415 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:07.933377    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:34:07.933514 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:07.933420    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls podName:9c3a8dde-f0f1-4ebe-88ce-f56cd784f813 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:11.933406684 +0000 UTC m=+42.591269135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls") pod "dns-default-6knhq" (UID: "9c3a8dde-f0f1-4ebe-88ce-f56cd784f813") : secret "dns-default-metrics-tls" not found
May 27 23:34:07.933514 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:07.933433    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert podName:8accd220-388c-4d68-b5d9-f393b8474848 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:11.933427095 +0000 UTC m=+42.591289547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert") pod "ingress-canary-hwcgw" (UID: "8accd220-388c-4d68-b5d9-f393b8474848") : secret "canary-serving-cert" not found
May 27 23:34:08.115877 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:08.115832    2562 generic.go:358] "Generic (PLEG): container finished" podID="7ebdbd61-602f-4e9b-95a1-ed43b4d13158" containerID="8a6c1c77a1e7e55fa4163e0cae2208a6991703ce137ac16fb4564f4c1a54c318" exitCode=0
May 27 23:34:08.116347 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:08.115905    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgd7k" event={"ID":"7ebdbd61-602f-4e9b-95a1-ed43b4d13158","Type":"ContainerDied","Data":"8a6c1c77a1e7e55fa4163e0cae2208a6991703ce137ac16fb4564f4c1a54c318"}
May 27 23:34:09.120603 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:09.120567    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgd7k" event={"ID":"7ebdbd61-602f-4e9b-95a1-ed43b4d13158","Type":"ContainerStarted","Data":"199aff64fd061cdfc99ac80cc08f750b4db81477683aa9d14431b7b4b2518262"}
May 27 23:34:09.144075 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:09.144028    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cgd7k" podStartSLOduration=5.600463624 podStartE2EDuration="39.144014862s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:33:32.571102742 +0000 UTC m=+3.228965194" lastFinishedPulling="2026-05-27 23:34:06.114653978 +0000 UTC m=+36.772516432" observedRunningTime="2026-05-27 23:34:09.143894236 +0000 UTC m=+39.801756710" watchObservedRunningTime="2026-05-27 23:34:09.144014862 +0000 UTC m=+39.801877336"
May 27 23:34:11.961520 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:11.961484    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:11.961893 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:11.961567    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:11.961893 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:11.961640    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:34:11.961893 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:11.961698    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls podName:9c3a8dde-f0f1-4ebe-88ce-f56cd784f813 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:19.961682815 +0000 UTC m=+50.619545266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls") pod "dns-default-6knhq" (UID: "9c3a8dde-f0f1-4ebe-88ce-f56cd784f813") : secret "dns-default-metrics-tls" not found
May 27 23:34:11.961893 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:11.961644    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:34:11.961893 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:11.961762    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert podName:8accd220-388c-4d68-b5d9-f393b8474848 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:19.961750905 +0000 UTC m=+50.619613356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert") pod "ingress-canary-hwcgw" (UID: "8accd220-388c-4d68-b5d9-f393b8474848") : secret "canary-serving-cert" not found
May 27 23:34:20.017093 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:20.017049    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:20.017560 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:20.017120    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:20.017560 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:20.017198    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:34:20.017560 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:20.017220    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:34:20.017560 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:20.017259    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert podName:8accd220-388c-4d68-b5d9-f393b8474848 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:36.017242732 +0000 UTC m=+66.675105184 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert") pod "ingress-canary-hwcgw" (UID: "8accd220-388c-4d68-b5d9-f393b8474848") : secret "canary-serving-cert" not found
May 27 23:34:20.017560 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:20.017272    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls podName:9c3a8dde-f0f1-4ebe-88ce-f56cd784f813 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:36.017266354 +0000 UTC m=+66.675128806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls") pod "dns-default-6knhq" (UID: "9c3a8dde-f0f1-4ebe-88ce-f56cd784f813") : secret "dns-default-metrics-tls" not found
May 27 23:34:29.102352 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:29.102317    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khn4g"
May 27 23:34:35.621928 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:35.621874    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:34:35.624790 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:35.624772    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 27 23:34:35.632203 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:35.632177    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 27 23:34:35.632279 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:35.632249    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs podName:c4abd5dc-33fe-4e84-a433-06f2916dab75 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:39.632227711 +0000 UTC m=+130.290090162 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs") pod "network-metrics-daemon-jd94m" (UID: "c4abd5dc-33fe-4e84-a433-06f2916dab75") : secret "metrics-daemon-secret" not found
May 27 23:34:35.722930 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:35.722895    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:34:35.726070 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:35.726054    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 27 23:34:35.736187 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:35.736168    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 27 23:34:35.746478 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:35.746451    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjqg\" (UniqueName: \"kubernetes.io/projected/c9d94568-264b-4508-9f30-a7bc58643904-kube-api-access-kkjqg\") pod \"network-check-target-qfvtb\" (UID: \"c9d94568-264b-4508-9f30-a7bc58643904\") " pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:34:35.855395 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:35.855365    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptj7k\""
May 27 23:34:35.863001 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:35.862970    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:34:36.014834 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:36.014801    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qfvtb"]
May 27 23:34:36.018644 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:34:36.018619    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d94568_264b_4508_9f30_a7bc58643904.slice/crio-f4a90db7831feed1f76ce66c301844d46125c5ef4a72556451e8db754017af28 WatchSource:0}: Error finding container f4a90db7831feed1f76ce66c301844d46125c5ef4a72556451e8db754017af28: Status 404 returned error can't find the container with id f4a90db7831feed1f76ce66c301844d46125c5ef4a72556451e8db754017af28
May 27 23:34:36.024352 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:36.024328    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:34:36.024432 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:36.024370    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:34:36.024470 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:36.024458    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:34:36.024470 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:36.024459    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:34:36.024554 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:36.024505    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls podName:9c3a8dde-f0f1-4ebe-88ce-f56cd784f813 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:08.024489885 +0000 UTC m=+98.682352337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls") pod "dns-default-6knhq" (UID: "9c3a8dde-f0f1-4ebe-88ce-f56cd784f813") : secret "dns-default-metrics-tls" not found
May 27 23:34:36.024554 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:36.024517    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert podName:8accd220-388c-4d68-b5d9-f393b8474848 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:08.024510973 +0000 UTC m=+98.682373425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert") pod "ingress-canary-hwcgw" (UID: "8accd220-388c-4d68-b5d9-f393b8474848") : secret "canary-serving-cert" not found
May 27 23:34:36.172086 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:36.171997    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qfvtb" event={"ID":"c9d94568-264b-4508-9f30-a7bc58643904","Type":"ContainerStarted","Data":"f4a90db7831feed1f76ce66c301844d46125c5ef4a72556451e8db754017af28"}
May 27 23:34:39.178763 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:39.178681    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qfvtb" event={"ID":"c9d94568-264b-4508-9f30-a7bc58643904","Type":"ContainerStarted","Data":"b26fdd614eafc1a809fccfb9d63f2f1b1fce09d9f8dd05e36019e3690f2ae7e3"}
May 27 23:34:39.179107 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:39.178812    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:34:39.196944 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:39.196901    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qfvtb" podStartSLOduration=66.420107122 podStartE2EDuration="1m9.196887548s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:34:36.020637219 +0000 UTC m=+66.678499672" lastFinishedPulling="2026-05-27 23:34:38.797417639 +0000 UTC m=+69.455280098" observedRunningTime="2026-05-27 23:34:39.19622865 +0000 UTC m=+69.854091124" watchObservedRunningTime="2026-05-27 23:34:39.196887548 +0000 UTC m=+69.854750021"
May 27 23:34:59.041244 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.041128    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-gtr8n"]
May 27 23:34:59.045348 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.045327    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-749c9c75f6-f7z9x"]
May 27 23:34:59.045516 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.045497    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.047936 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.047917    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.048382 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.048363    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 27 23:34:59.048586 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.048558    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 27 23:34:59.048708 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.048592    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 27 23:34:59.048708 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.048636    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qb9mz\""
May 27 23:34:59.048708 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.048568    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 27 23:34:59.051456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.051439    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 27 23:34:59.051585 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.051571    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 27 23:34:59.051653 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.051590    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 27 23:34:59.051653 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.051629    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 27 23:34:59.051755 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.051696    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 27 23:34:59.051846 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.051819    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 27 23:34:59.052833 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.052811    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-gtr8n"]
May 27 23:34:59.053125 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.053107    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p2k2z\""
May 27 23:34:59.056420 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.056401    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 27 23:34:59.058785 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.058763    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-749c9c75f6-f7z9x"]
May 27 23:34:59.085953 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.085913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56d6868c-dba0-46ce-aaa0-78324081e3e4-tmp\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.086124 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56d6868c-dba0-46ce-aaa0-78324081e3e4-serving-cert\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.086124 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvzvt\" (UniqueName: \"kubernetes.io/projected/56d6868c-dba0-46ce-aaa0-78324081e3e4-kube-api-access-mvzvt\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.086124 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-stats-auth\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.086275 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/56d6868c-dba0-46ce-aaa0-78324081e3e4-snapshots\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.086275 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d6868c-dba0-46ce-aaa0-78324081e3e4-service-ca-bundle\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.086275 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.086390 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.086390 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-default-certificate\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.086390 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdsr\" (UniqueName: \"kubernetes.io/projected/303186fe-da16-4eef-9f9b-6406b1ae5d04-kube-api-access-xhdsr\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.086390 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.086385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d6868c-dba0-46ce-aaa0-78324081e3e4-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.186938 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.186901    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d6868c-dba0-46ce-aaa0-78324081e3e4-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.187121 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.186956    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56d6868c-dba0-46ce-aaa0-78324081e3e4-tmp\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.187121 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.186997    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56d6868c-dba0-46ce-aaa0-78324081e3e4-serving-cert\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.187121 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187022    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mvzvt\" (UniqueName: \"kubernetes.io/projected/56d6868c-dba0-46ce-aaa0-78324081e3e4-kube-api-access-mvzvt\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.187121 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187048    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-stats-auth\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.187345 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187314    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/56d6868c-dba0-46ce-aaa0-78324081e3e4-snapshots\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.187495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187366    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d6868c-dba0-46ce-aaa0-78324081e3e4-service-ca-bundle\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.187495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187397    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.187495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187425    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56d6868c-dba0-46ce-aaa0-78324081e3e4-tmp\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.187495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187432    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.187495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187488    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-default-certificate\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.187802 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187555    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdsr\" (UniqueName: \"kubernetes.io/projected/303186fe-da16-4eef-9f9b-6406b1ae5d04-kube-api-access-xhdsr\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.187802 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:59.187587    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:59.687564832 +0000 UTC m=+90.345427284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : configmap references non-existent config key: service-ca.crt
May 27 23:34:59.187802 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:59.187772    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 27 23:34:59.187978 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:59.187812    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:34:59.687799411 +0000 UTC m=+90.345661879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : secret "router-metrics-certs-default" not found
May 27 23:34:59.188043 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.187890    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/56d6868c-dba0-46ce-aaa0-78324081e3e4-snapshots\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.191930 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.188291    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d6868c-dba0-46ce-aaa0-78324081e3e4-service-ca-bundle\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.191930 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.190804    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-default-certificate\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.192103 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.192048    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d6868c-dba0-46ce-aaa0-78324081e3e4-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.192600 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.192577    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56d6868c-dba0-46ce-aaa0-78324081e3e4-serving-cert\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.192684 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.192632    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-stats-auth\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.198450 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.198419    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvzvt\" (UniqueName: \"kubernetes.io/projected/56d6868c-dba0-46ce-aaa0-78324081e3e4-kube-api-access-mvzvt\") pod \"insights-operator-78bdfb888c-gtr8n\" (UID: \"56d6868c-dba0-46ce-aaa0-78324081e3e4\") " pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.198600 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.198582    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdsr\" (UniqueName: \"kubernetes.io/projected/303186fe-da16-4eef-9f9b-6406b1ae5d04-kube-api-access-xhdsr\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.356127 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.356027    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-gtr8n"
May 27 23:34:59.494909 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.494869    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-gtr8n"]
May 27 23:34:59.498558 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:34:59.498520    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d6868c_dba0_46ce_aaa0_78324081e3e4.slice/crio-f83d68f74db5d06b57d117d0f851a81af689dd90341122bcb217928f9a6bb399 WatchSource:0}: Error finding container f83d68f74db5d06b57d117d0f851a81af689dd90341122bcb217928f9a6bb399: Status 404 returned error can't find the container with id f83d68f74db5d06b57d117d0f851a81af689dd90341122bcb217928f9a6bb399
May 27 23:34:59.692123 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.692040    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.692123 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:34:59.692080    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:34:59.692292 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:59.692188    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 27 23:34:59.692292 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:59.692224    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:00.692211086 +0000 UTC m=+91.350073537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : configmap references non-existent config key: service-ca.crt
May 27 23:34:59.692292 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:34:59.692240    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:00.692233323 +0000 UTC m=+91.350095774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : secret "router-metrics-certs-default" not found
May 27 23:35:00.217875 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:00.217842    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-gtr8n" event={"ID":"56d6868c-dba0-46ce-aaa0-78324081e3e4","Type":"ContainerStarted","Data":"f83d68f74db5d06b57d117d0f851a81af689dd90341122bcb217928f9a6bb399"}
May 27 23:35:00.700755 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:00.700720    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:00.700926 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:00.700761    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:00.700926 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:00.700920    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 27 23:35:00.701025 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:00.700922    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:02.70090363 +0000 UTC m=+93.358766085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : configmap references non-existent config key: service-ca.crt
May 27 23:35:00.701025 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:00.700969    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:02.70095938 +0000 UTC m=+93.358821834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : secret "router-metrics-certs-default" not found
May 27 23:35:02.052071 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.052038    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh"]
May 27 23:35:02.055039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.055018    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh"
May 27 23:35:02.058074 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.058048    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 27 23:35:02.059040 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.059022    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 27 23:35:02.059118 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.059072    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ltgv2\""
May 27 23:35:02.063621 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.063590    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh"]
May 27 23:35:02.111858 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.111821    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p574\" (UniqueName: \"kubernetes.io/projected/8c1ef9e8-d3be-4a0b-b38d-b96a11289ee5-kube-api-access-4p574\") pod \"volume-data-source-validator-5cf8c89d64-mstjh\" (UID: \"8c1ef9e8-d3be-4a0b-b38d-b96a11289ee5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh"
May 27 23:35:02.156510 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.156478    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"]
May 27 23:35:02.159355 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.159340    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:02.162086 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.162069    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 27 23:35:02.162483 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.162467    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 27 23:35:02.162598 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.162480    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 27 23:35:02.162598 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.162513    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z49wf\""
May 27 23:35:02.171735 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.171701    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"]
May 27 23:35:02.212682 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.212652    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4p574\" (UniqueName: \"kubernetes.io/projected/8c1ef9e8-d3be-4a0b-b38d-b96a11289ee5-kube-api-access-4p574\") pod \"volume-data-source-validator-5cf8c89d64-mstjh\" (UID: \"8c1ef9e8-d3be-4a0b-b38d-b96a11289ee5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh"
May 27 23:35:02.212682 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.212689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg4rb\" (UniqueName: \"kubernetes.io/projected/61e303c0-6b4f-437a-8f22-17e89a23c62b-kube-api-access-cg4rb\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:02.212897 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.212741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:02.222428 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.222403    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p574\" (UniqueName: \"kubernetes.io/projected/8c1ef9e8-d3be-4a0b-b38d-b96a11289ee5-kube-api-access-4p574\") pod \"volume-data-source-validator-5cf8c89d64-mstjh\" (UID: \"8c1ef9e8-d3be-4a0b-b38d-b96a11289ee5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh"
May 27 23:35:02.223394 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.223360    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-gtr8n" event={"ID":"56d6868c-dba0-46ce-aaa0-78324081e3e4","Type":"ContainerStarted","Data":"aed330a283a4bf6f4f6e26f75af8dd8f30e412fe02fab94d1191299c7780f637"}
May 27 23:35:02.243701 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.243639    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-gtr8n" podStartSLOduration=1.267194569 podStartE2EDuration="3.243617265s" podCreationTimestamp="2026-05-27 23:34:59 +0000 UTC" firstStartedPulling="2026-05-27 23:34:59.500189873 +0000 UTC m=+90.158052325" lastFinishedPulling="2026-05-27 23:35:01.476612568 +0000 UTC m=+92.134475021" observedRunningTime="2026-05-27 23:35:02.242549363 +0000 UTC m=+92.900411838" watchObservedRunningTime="2026-05-27 23:35:02.243617265 +0000 UTC m=+92.901479740"
May 27 23:35:02.314025 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.313928    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cg4rb\" (UniqueName: \"kubernetes.io/projected/61e303c0-6b4f-437a-8f22-17e89a23c62b-kube-api-access-cg4rb\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:02.314207 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.314114    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:02.314274 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:02.314231    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:35:02.314336 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:02.314313    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls podName:61e303c0-6b4f-437a-8f22-17e89a23c62b nodeName:}" failed. No retries permitted until 2026-05-27 23:35:02.814290109 +0000 UTC m=+93.472152563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls") pod "cluster-samples-operator-697587599-2rgz7" (UID: "61e303c0-6b4f-437a-8f22-17e89a23c62b") : secret "samples-operator-tls" not found
May 27 23:35:02.323715 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.323684    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg4rb\" (UniqueName: \"kubernetes.io/projected/61e303c0-6b4f-437a-8f22-17e89a23c62b-kube-api-access-cg4rb\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:02.364410 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.364376    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh"
May 27 23:35:02.491750 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.491719    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh"]
May 27 23:35:02.494672 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:02.494644    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1ef9e8_d3be_4a0b_b38d_b96a11289ee5.slice/crio-65cab5341c33203edba0f446d8dcf31f7cdd58b599f1bdb9bee55adddc41f23d WatchSource:0}: Error finding container 65cab5341c33203edba0f446d8dcf31f7cdd58b599f1bdb9bee55adddc41f23d: Status 404 returned error can't find the container with id 65cab5341c33203edba0f446d8dcf31f7cdd58b599f1bdb9bee55adddc41f23d
May 27 23:35:02.717282 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.717190    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:02.717282 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.717231    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:02.717463 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:02.717339    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 27 23:35:02.717463 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:02.717406    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:06.717387604 +0000 UTC m=+97.375250056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : secret "router-metrics-certs-default" not found
May 27 23:35:02.717463 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:02.717420    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:06.717413465 +0000 UTC m=+97.375275917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : configmap references non-existent config key: service-ca.crt
May 27 23:35:02.817615 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:02.817578    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:02.817810 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:02.817698    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:35:02.817810 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:02.817782    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls podName:61e303c0-6b4f-437a-8f22-17e89a23c62b nodeName:}" failed. No retries permitted until 2026-05-27 23:35:03.817762656 +0000 UTC m=+94.475625109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls") pod "cluster-samples-operator-697587599-2rgz7" (UID: "61e303c0-6b4f-437a-8f22-17e89a23c62b") : secret "samples-operator-tls" not found
May 27 23:35:03.226546 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:03.226490    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh" event={"ID":"8c1ef9e8-d3be-4a0b-b38d-b96a11289ee5","Type":"ContainerStarted","Data":"65cab5341c33203edba0f446d8dcf31f7cdd58b599f1bdb9bee55adddc41f23d"}
May 27 23:35:03.826330 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:03.826270    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:03.826524 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:03.826408    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:35:03.826524 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:03.826487    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls podName:61e303c0-6b4f-437a-8f22-17e89a23c62b nodeName:}" failed. No retries permitted until 2026-05-27 23:35:05.826465769 +0000 UTC m=+96.484328226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls") pod "cluster-samples-operator-697587599-2rgz7" (UID: "61e303c0-6b4f-437a-8f22-17e89a23c62b") : secret "samples-operator-tls" not found
May 27 23:35:04.022911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.022872    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-rgfxl"]
May 27 23:35:04.025825 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.025801    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.029549 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.029502    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 27 23:35:04.029663 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.029504    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 27 23:35:04.029663 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.029565    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 27 23:35:04.029663 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.029504    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xttzb\""
May 27 23:35:04.029994 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.029973    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 27 23:35:04.034289 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.034262    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 27 23:35:04.035103 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.035084    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-rgfxl"]
May 27 23:35:04.129376 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.129287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxnw\" (UniqueName: \"kubernetes.io/projected/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-kube-api-access-7nxnw\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.129376 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.129340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-config\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.129627 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.129455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-trusted-ca\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.129627 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.129508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-serving-cert\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.229407 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.229369    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh" event={"ID":"8c1ef9e8-d3be-4a0b-b38d-b96a11289ee5","Type":"ContainerStarted","Data":"96a90630e89a303a7a357eba84bb0888065b66e48208e219f33c88bbc683588d"}
May 27 23:35:04.229874 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.229855    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxnw\" (UniqueName: \"kubernetes.io/projected/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-kube-api-access-7nxnw\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.229935 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.229892    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-config\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.229935 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.229915    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-trusted-ca\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.230077 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.230059    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-serving-cert\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.230477 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.230452    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-config\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.230744 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.230726    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-trusted-ca\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.232297 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.232278    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-serving-cert\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.238193 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.238165    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxnw\" (UniqueName: \"kubernetes.io/projected/7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c-kube-api-access-7nxnw\") pod \"console-operator-76dff48d59-rgfxl\" (UID: \"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c\") " pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.245640 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.245599    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-mstjh" podStartSLOduration=0.830194729 podStartE2EDuration="2.245586585s" podCreationTimestamp="2026-05-27 23:35:02 +0000 UTC" firstStartedPulling="2026-05-27 23:35:02.496430141 +0000 UTC m=+93.154292592" lastFinishedPulling="2026-05-27 23:35:03.911821993 +0000 UTC m=+94.569684448" observedRunningTime="2026-05-27 23:35:04.245006699 +0000 UTC m=+94.902869172" watchObservedRunningTime="2026-05-27 23:35:04.245586585 +0000 UTC m=+94.903449093"
May 27 23:35:04.335830 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.335790    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:04.461409 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.460971    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-rgfxl"]
May 27 23:35:04.598513 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:04.598484    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dbcvm_1f795878-59a3-488e-af3e-9df4e143020a/dns-node-resolver/0.log"
May 27 23:35:05.232423 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:05.232389    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl" event={"ID":"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c","Type":"ContainerStarted","Data":"166285fb9929c34cdeedc6e22f17c38870967e06a7c068de73dd00828f81c7dc"}
May 27 23:35:05.598788 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:05.598762    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j7qvm_4a1534cf-d950-40c0-9a7e-18fc4fc88ee0/node-ca/0.log"
May 27 23:35:05.842346 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:05.842307    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:05.842559 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:05.842462    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:35:05.842559 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:05.842550    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls podName:61e303c0-6b4f-437a-8f22-17e89a23c62b nodeName:}" failed. No retries permitted until 2026-05-27 23:35:09.842514142 +0000 UTC m=+100.500376599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls") pod "cluster-samples-operator-697587599-2rgz7" (UID: "61e303c0-6b4f-437a-8f22-17e89a23c62b") : secret "samples-operator-tls" not found
May 27 23:35:06.749018 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:06.748957    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:06.749500 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:06.749120    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:06.749500 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:06.749139    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:14.749120439 +0000 UTC m=+105.406982911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : configmap references non-existent config key: service-ca.crt
May 27 23:35:06.749500 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:06.749216    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 27 23:35:06.749500 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:06.749268    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:14.749250201 +0000 UTC m=+105.407112654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : secret "router-metrics-certs-default" not found
May 27 23:35:07.237335 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:07.237260    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/0.log"
May 27 23:35:07.237335 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:07.237297    2562 generic.go:358] "Generic (PLEG): container finished" podID="7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c" containerID="ce27928aadbc289ef415ffade1c24110b2884c9fa39f8d59008fb9f1b874763c" exitCode=255
May 27 23:35:07.237519 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:07.237349    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl" event={"ID":"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c","Type":"ContainerDied","Data":"ce27928aadbc289ef415ffade1c24110b2884c9fa39f8d59008fb9f1b874763c"}
May 27 23:35:07.237631 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:07.237616    2562 scope.go:117] "RemoveContainer" containerID="ce27928aadbc289ef415ffade1c24110b2884c9fa39f8d59008fb9f1b874763c"
May 27 23:35:08.059248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:08.059206    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:35:08.059652 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:08.059273    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:35:08.059652 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:08.059356    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:35:08.059652 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:08.059359    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:35:08.059652 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:08.059409    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls podName:9c3a8dde-f0f1-4ebe-88ce-f56cd784f813 nodeName:}" failed. No retries permitted until 2026-05-27 23:36:12.059393909 +0000 UTC m=+162.717256374 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls") pod "dns-default-6knhq" (UID: "9c3a8dde-f0f1-4ebe-88ce-f56cd784f813") : secret "dns-default-metrics-tls" not found
May 27 23:35:08.059652 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:08.059422    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert podName:8accd220-388c-4d68-b5d9-f393b8474848 nodeName:}" failed. No retries permitted until 2026-05-27 23:36:12.059415299 +0000 UTC m=+162.717277750 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert") pod "ingress-canary-hwcgw" (UID: "8accd220-388c-4d68-b5d9-f393b8474848") : secret "canary-serving-cert" not found
May 27 23:35:08.242370 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:08.242345    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:35:08.242741 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:08.242724    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/0.log"
May 27 23:35:08.242846 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:08.242764    2562 generic.go:358] "Generic (PLEG): container finished" podID="7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c" containerID="0029e754e4289bb5c63c51f43c83cf9a3172af44099218fd5f3abc216f238d51" exitCode=255
May 27 23:35:08.242846 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:08.242813    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl" event={"ID":"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c","Type":"ContainerDied","Data":"0029e754e4289bb5c63c51f43c83cf9a3172af44099218fd5f3abc216f238d51"}
May 27 23:35:08.242944 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:08.242851    2562 scope.go:117] "RemoveContainer" containerID="ce27928aadbc289ef415ffade1c24110b2884c9fa39f8d59008fb9f1b874763c"
May 27 23:35:08.243114 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:08.243090    2562 scope.go:117] "RemoveContainer" containerID="0029e754e4289bb5c63c51f43c83cf9a3172af44099218fd5f3abc216f238d51"
May 27 23:35:08.243297 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:08.243275    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-rgfxl_openshift-console-operator(7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c)\"" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl" podUID="7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c"
May 27 23:35:09.245622 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:09.245591    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:35:09.246048 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:09.245918    2562 scope.go:117] "RemoveContainer" containerID="0029e754e4289bb5c63c51f43c83cf9a3172af44099218fd5f3abc216f238d51"
May 27 23:35:09.246101 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:09.246074    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-rgfxl_openshift-console-operator(7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c)\"" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl" podUID="7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c"
May 27 23:35:09.872443 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:09.872402    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:09.872658 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:09.872573    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:35:09.872658 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:09.872638    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls podName:61e303c0-6b4f-437a-8f22-17e89a23c62b nodeName:}" failed. No retries permitted until 2026-05-27 23:35:17.872623359 +0000 UTC m=+108.530485811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls") pod "cluster-samples-operator-697587599-2rgz7" (UID: "61e303c0-6b4f-437a-8f22-17e89a23c62b") : secret "samples-operator-tls" not found
May 27 23:35:10.182962 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.182887    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qfvtb"
May 27 23:35:10.477828 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.477754    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5"]
May 27 23:35:10.480431 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.480415    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5"
May 27 23:35:10.484540 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.484476    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pb8zd\""
May 27 23:35:10.492077 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.492051    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5"]
May 27 23:35:10.577216 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.577180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lj5k\" (UniqueName: \"kubernetes.io/projected/bdc00c39-ffab-4200-9e50-bb654d6d6463-kube-api-access-6lj5k\") pod \"network-check-source-7f44b6dd87-fd8x5\" (UID: \"bdc00c39-ffab-4200-9e50-bb654d6d6463\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5"
May 27 23:35:10.677723 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.677673    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6lj5k\" (UniqueName: \"kubernetes.io/projected/bdc00c39-ffab-4200-9e50-bb654d6d6463-kube-api-access-6lj5k\") pod \"network-check-source-7f44b6dd87-fd8x5\" (UID: \"bdc00c39-ffab-4200-9e50-bb654d6d6463\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5"
May 27 23:35:10.686071 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.686048    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lj5k\" (UniqueName: \"kubernetes.io/projected/bdc00c39-ffab-4200-9e50-bb654d6d6463-kube-api-access-6lj5k\") pod \"network-check-source-7f44b6dd87-fd8x5\" (UID: \"bdc00c39-ffab-4200-9e50-bb654d6d6463\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5"
May 27 23:35:10.788492 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.788460    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5"
May 27 23:35:10.911422 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:10.911386    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5"]
May 27 23:35:10.914212 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:10.914187    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc00c39_ffab_4200_9e50_bb654d6d6463.slice/crio-6b22fcff493f8cf5b66dd4cd5bc5d39cc08fcd5c8cb394286fb7c5dc5ff98dc6 WatchSource:0}: Error finding container 6b22fcff493f8cf5b66dd4cd5bc5d39cc08fcd5c8cb394286fb7c5dc5ff98dc6: Status 404 returned error can't find the container with id 6b22fcff493f8cf5b66dd4cd5bc5d39cc08fcd5c8cb394286fb7c5dc5ff98dc6
May 27 23:35:11.252049 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:11.252013    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5" event={"ID":"bdc00c39-ffab-4200-9e50-bb654d6d6463","Type":"ContainerStarted","Data":"5575154e31ef49eb9959cbf1e08638a7a9977aad9bbb0102a0f27a2543a59f3b"}
May 27 23:35:11.252210 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:11.252054    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5" event={"ID":"bdc00c39-ffab-4200-9e50-bb654d6d6463","Type":"ContainerStarted","Data":"6b22fcff493f8cf5b66dd4cd5bc5d39cc08fcd5c8cb394286fb7c5dc5ff98dc6"}
May 27 23:35:11.273076 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:11.273031    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-fd8x5" podStartSLOduration=1.273015972 podStartE2EDuration="1.273015972s" podCreationTimestamp="2026-05-27 23:35:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:35:11.272289342 +0000 UTC m=+101.930151819" watchObservedRunningTime="2026-05-27 23:35:11.273015972 +0000 UTC m=+101.930878464"
May 27 23:35:14.336280 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:14.336239    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:14.336667 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:14.336291    2562 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:14.336667 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:14.336652    2562 scope.go:117] "RemoveContainer" containerID="0029e754e4289bb5c63c51f43c83cf9a3172af44099218fd5f3abc216f238d51"
May 27 23:35:14.336833 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:14.336815    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-rgfxl_openshift-console-operator(7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c)\"" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl" podUID="7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c"
May 27 23:35:14.805754 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:14.805714    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:14.805936 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:14.805765    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:14.805936 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:14.805865    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 27 23:35:14.805936 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:14.805915    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:30.805897013 +0000 UTC m=+121.463759465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : configmap references non-existent config key: service-ca.crt
May 27 23:35:14.805936 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:14.805930    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs podName:303186fe-da16-4eef-9f9b-6406b1ae5d04 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:30.805924287 +0000 UTC m=+121.463786740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs") pod "router-default-749c9c75f6-f7z9x" (UID: "303186fe-da16-4eef-9f9b-6406b1ae5d04") : secret "router-metrics-certs-default" not found
May 27 23:35:17.933341 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:17.933296    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:17.935653 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:17.935629    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61e303c0-6b4f-437a-8f22-17e89a23c62b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-2rgz7\" (UID: \"61e303c0-6b4f-437a-8f22-17e89a23c62b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:18.072091 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:18.072036    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"
May 27 23:35:18.200166 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:18.200082    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7"]
May 27 23:35:18.268718 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:18.268685    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7" event={"ID":"61e303c0-6b4f-437a-8f22-17e89a23c62b","Type":"ContainerStarted","Data":"d4703e16124e744a3b2b9b091268c60b62eaa5bdff0721fd7b46d23687d62fe9"}
May 27 23:35:20.275958 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:20.275922    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7" event={"ID":"61e303c0-6b4f-437a-8f22-17e89a23c62b","Type":"ContainerStarted","Data":"3681f5029a955b9d154b48ebeb20099723aa2b06c4ddbf3d3fba95ad7ea4ba64"}
May 27 23:35:20.275958 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:20.275958    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7" event={"ID":"61e303c0-6b4f-437a-8f22-17e89a23c62b","Type":"ContainerStarted","Data":"46b6108a1b2fb12b1094001a97a0e88bdd434982912f8dca616de4cbb9cbc02a"}
May 27 23:35:20.292068 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:20.292013    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-2rgz7" podStartSLOduration=16.793565117 podStartE2EDuration="18.291999547s" podCreationTimestamp="2026-05-27 23:35:02 +0000 UTC" firstStartedPulling="2026-05-27 23:35:18.24286493 +0000 UTC m=+108.900727386" lastFinishedPulling="2026-05-27 23:35:19.741299363 +0000 UTC m=+110.399161816" observedRunningTime="2026-05-27 23:35:20.291744508 +0000 UTC m=+110.949606993" watchObservedRunningTime="2026-05-27 23:35:20.291999547 +0000 UTC m=+110.949862020"
May 27 23:35:27.940891 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:27.940852    2562 scope.go:117] "RemoveContainer" containerID="0029e754e4289bb5c63c51f43c83cf9a3172af44099218fd5f3abc216f238d51"
May 27 23:35:28.295328 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:28.295301    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:35:28.295502 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:28.295374    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl" event={"ID":"7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c","Type":"ContainerStarted","Data":"0abf10e548c66583285c553941fea747aff809292f65a1d212b3e977dc2736c8"}
May 27 23:35:28.295657 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:28.295635    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:28.314275 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:28.314211    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl" podStartSLOduration=22.573100831 podStartE2EDuration="24.31419097s" podCreationTimestamp="2026-05-27 23:35:04 +0000 UTC" firstStartedPulling="2026-05-27 23:35:04.465134144 +0000 UTC m=+95.122996596" lastFinishedPulling="2026-05-27 23:35:06.206224273 +0000 UTC m=+96.864086735" observedRunningTime="2026-05-27 23:35:28.313221238 +0000 UTC m=+118.971083712" watchObservedRunningTime="2026-05-27 23:35:28.31419097 +0000 UTC m=+118.972053446"
May 27 23:35:28.406623 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:28.406593    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-rgfxl"
May 27 23:35:30.836130 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:30.836091    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:30.836130 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:30.836133    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:30.836695 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:30.836677    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/303186fe-da16-4eef-9f9b-6406b1ae5d04-service-ca-bundle\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:30.838397 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:30.838373    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303186fe-da16-4eef-9f9b-6406b1ae5d04-metrics-certs\") pod \"router-default-749c9c75f6-f7z9x\" (UID: \"303186fe-da16-4eef-9f9b-6406b1ae5d04\") " pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:30.865836 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:30.865806    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p2k2z\""
May 27 23:35:30.873917 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:30.873886    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:31.000613 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:31.000580    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-749c9c75f6-f7z9x"]
May 27 23:35:31.003479 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:31.003452    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303186fe_da16_4eef_9f9b_6406b1ae5d04.slice/crio-e60ee1b5f508ceb68cd263cd33e11ac55eab8c2cc9d7dfd00688b4820241179e WatchSource:0}: Error finding container e60ee1b5f508ceb68cd263cd33e11ac55eab8c2cc9d7dfd00688b4820241179e: Status 404 returned error can't find the container with id e60ee1b5f508ceb68cd263cd33e11ac55eab8c2cc9d7dfd00688b4820241179e
May 27 23:35:31.304374 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:31.304337    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-749c9c75f6-f7z9x" event={"ID":"303186fe-da16-4eef-9f9b-6406b1ae5d04","Type":"ContainerStarted","Data":"0847464e4faa07ae97ea7b83988c9c65982e86aafe5b58f5e6be542129811a1e"}
May 27 23:35:31.304374 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:31.304376    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-749c9c75f6-f7z9x" event={"ID":"303186fe-da16-4eef-9f9b-6406b1ae5d04","Type":"ContainerStarted","Data":"e60ee1b5f508ceb68cd263cd33e11ac55eab8c2cc9d7dfd00688b4820241179e"}
May 27 23:35:31.324959 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:31.324911    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-749c9c75f6-f7z9x" podStartSLOduration=32.324896488 podStartE2EDuration="32.324896488s" podCreationTimestamp="2026-05-27 23:34:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:35:31.324164174 +0000 UTC m=+121.982026647" watchObservedRunningTime="2026-05-27 23:35:31.324896488 +0000 UTC m=+121.982758961"
May 27 23:35:31.874840 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:31.874795    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:31.877256 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:31.877235    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:32.306771 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:32.306742    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:32.307994 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:32.307972    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-749c9c75f6-f7z9x"
May 27 23:35:35.823255 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.823222    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"]
May 27 23:35:35.827697 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.827675    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"
May 27 23:35:35.831791 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.831765    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9vxmd\""
May 27 23:35:35.831791 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.831780    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 27 23:35:35.832015 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.831770    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 27 23:35:35.843604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.843584    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"]
May 27 23:35:35.927667 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.927634    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fdf79cb5c-hwdg9"]
May 27 23:35:35.930763 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.930738    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wvmmg"]
May 27 23:35:35.930896 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.930879    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:35.933867 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.933850    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:35.942714 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.942692    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 27 23:35:35.943222 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.942810    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 27 23:35:35.943222 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.943078    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 27 23:35:35.943343 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.943309    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xkpcw\""
May 27 23:35:35.943394 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.943309    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 27 23:35:35.943394 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.943367    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dkbtx\""
May 27 23:35:35.944128 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.944111    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 27 23:35:35.944785 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.944770    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 27 23:35:35.944957 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.944943    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 27 23:35:35.945999 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.945983    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 27 23:35:35.946096 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.946082    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 27 23:35:35.951476 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.951456    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fdf79cb5c-hwdg9"]
May 27 23:35:35.969671 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.969640    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvmmg"]
May 27 23:35:35.972137 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.972111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dfee3cde-baf8-4bb9-9a6c-992c1b43e77a-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-vznvd\" (UID: \"dfee3cde-baf8-4bb9-9a6c-992c1b43e77a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"
May 27 23:35:35.972307 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:35.972152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dfee3cde-baf8-4bb9-9a6c-992c1b43e77a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-vznvd\" (UID: \"dfee3cde-baf8-4bb9-9a6c-992c1b43e77a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"
May 27 23:35:36.073167 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073130    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dfee3cde-baf8-4bb9-9a6c-992c1b43e77a-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-vznvd\" (UID: \"dfee3cde-baf8-4bb9-9a6c-992c1b43e77a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"
May 27 23:35:36.073368 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-config\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.073368 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073248    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558pn\" (UniqueName: \"kubernetes.io/projected/feb33324-55b0-422e-85f3-2ad64b45c12d-kube-api-access-558pn\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.073368 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-oauth-serving-cert\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.073540 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073382    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/feb33324-55b0-422e-85f3-2ad64b45c12d-data-volume\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.073540 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/feb33324-55b0-422e-85f3-2ad64b45c12d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.073540 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073435    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dfee3cde-baf8-4bb9-9a6c-992c1b43e77a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-vznvd\" (UID: \"dfee3cde-baf8-4bb9-9a6c-992c1b43e77a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"
May 27 23:35:36.073540 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073477    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/feb33324-55b0-422e-85f3-2ad64b45c12d-crio-socket\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.073540 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073509    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzjm\" (UniqueName: \"kubernetes.io/projected/ac05b693-5665-41ba-8647-9b49ad5e7ebe-kube-api-access-bxzjm\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.073792 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-serving-cert\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.073792 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/feb33324-55b0-422e-85f3-2ad64b45c12d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.073792 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-service-ca\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.073792 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-oauth-config\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.073792 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.073786    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dfee3cde-baf8-4bb9-9a6c-992c1b43e77a-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-vznvd\" (UID: \"dfee3cde-baf8-4bb9-9a6c-992c1b43e77a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"
May 27 23:35:36.075873 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.075849    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dfee3cde-baf8-4bb9-9a6c-992c1b43e77a-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-vznvd\" (UID: \"dfee3cde-baf8-4bb9-9a6c-992c1b43e77a\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"
May 27 23:35:36.137544 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.137499    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"
May 27 23:35:36.174640 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174608    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-config\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.174817 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174670    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-558pn\" (UniqueName: \"kubernetes.io/projected/feb33324-55b0-422e-85f3-2ad64b45c12d-kube-api-access-558pn\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.174817 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174708    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-oauth-serving-cert\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.174817 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174742    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/feb33324-55b0-422e-85f3-2ad64b45c12d-data-volume\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.174817 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174770    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/feb33324-55b0-422e-85f3-2ad64b45c12d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.174817 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174799    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/feb33324-55b0-422e-85f3-2ad64b45c12d-crio-socket\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.175076 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174823    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzjm\" (UniqueName: \"kubernetes.io/projected/ac05b693-5665-41ba-8647-9b49ad5e7ebe-kube-api-access-bxzjm\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.175076 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174850    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-serving-cert\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.175076 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174897    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/feb33324-55b0-422e-85f3-2ad64b45c12d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.175076 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174925    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-service-ca\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.175076 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.174960    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-oauth-config\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.175317 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.175107    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/feb33324-55b0-422e-85f3-2ad64b45c12d-data-volume\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.175317 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.175215    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-config\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.175317 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.175269    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/feb33324-55b0-422e-85f3-2ad64b45c12d-crio-socket\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.175471 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.175413    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-oauth-serving-cert\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.175661 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.175619    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/feb33324-55b0-422e-85f3-2ad64b45c12d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.175876 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.175859    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-service-ca\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.177864 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.177845    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-serving-cert\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.177864 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.177859    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-oauth-config\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.178010 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.177988    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/feb33324-55b0-422e-85f3-2ad64b45c12d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.184979 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.184616    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-558pn\" (UniqueName: \"kubernetes.io/projected/feb33324-55b0-422e-85f3-2ad64b45c12d-kube-api-access-558pn\") pod \"insights-runtime-extractor-wvmmg\" (UID: \"feb33324-55b0-422e-85f3-2ad64b45c12d\") " pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.185281 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.185258    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzjm\" (UniqueName: \"kubernetes.io/projected/ac05b693-5665-41ba-8647-9b49ad5e7ebe-kube-api-access-bxzjm\") pod \"console-6fdf79cb5c-hwdg9\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") " pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.240820 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.240792    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:36.246719 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.246697    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvmmg"
May 27 23:35:36.261057 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.261032    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd"]
May 27 23:35:36.263677 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:36.263653    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfee3cde_baf8_4bb9_9a6c_992c1b43e77a.slice/crio-0014d9bbb1b7b484475dc2e530588ad538e53b310182df8e291a81e71e4569a9 WatchSource:0}: Error finding container 0014d9bbb1b7b484475dc2e530588ad538e53b310182df8e291a81e71e4569a9: Status 404 returned error can't find the container with id 0014d9bbb1b7b484475dc2e530588ad538e53b310182df8e291a81e71e4569a9
May 27 23:35:36.318746 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.318714    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd" event={"ID":"dfee3cde-baf8-4bb9-9a6c-992c1b43e77a","Type":"ContainerStarted","Data":"0014d9bbb1b7b484475dc2e530588ad538e53b310182df8e291a81e71e4569a9"}
May 27 23:35:36.372864 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.372831    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fdf79cb5c-hwdg9"]
May 27 23:35:36.376620 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:36.376594    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac05b693_5665_41ba_8647_9b49ad5e7ebe.slice/crio-40d72a8fba8523b57db89a707e928272ed824bf2950d4a14fb455155fb862557 WatchSource:0}: Error finding container 40d72a8fba8523b57db89a707e928272ed824bf2950d4a14fb455155fb862557: Status 404 returned error can't find the container with id 40d72a8fba8523b57db89a707e928272ed824bf2950d4a14fb455155fb862557
May 27 23:35:36.390319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:36.390295    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvmmg"]
May 27 23:35:36.392793 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:36.392767    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb33324_55b0_422e_85f3_2ad64b45c12d.slice/crio-a9e2af2fc2f8e87c3717b931655b59f22bd8ffe8356351833b83a373b78ad2d4 WatchSource:0}: Error finding container a9e2af2fc2f8e87c3717b931655b59f22bd8ffe8356351833b83a373b78ad2d4: Status 404 returned error can't find the container with id a9e2af2fc2f8e87c3717b931655b59f22bd8ffe8356351833b83a373b78ad2d4
May 27 23:35:37.323739 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:37.323703    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvmmg" event={"ID":"feb33324-55b0-422e-85f3-2ad64b45c12d","Type":"ContainerStarted","Data":"5e2204c6fdaca228644e74dbc84bfc9291e6d9e9e570406191855477c3130606"}
May 27 23:35:37.323739 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:37.323743    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvmmg" event={"ID":"feb33324-55b0-422e-85f3-2ad64b45c12d","Type":"ContainerStarted","Data":"6b36ee900a7148599cb72a0b596729f04216141c9b0efee82e30215136c408b5"}
May 27 23:35:37.324264 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:37.323754    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvmmg" event={"ID":"feb33324-55b0-422e-85f3-2ad64b45c12d","Type":"ContainerStarted","Data":"a9e2af2fc2f8e87c3717b931655b59f22bd8ffe8356351833b83a373b78ad2d4"}
May 27 23:35:37.324892 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:37.324862    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdf79cb5c-hwdg9" event={"ID":"ac05b693-5665-41ba-8647-9b49ad5e7ebe","Type":"ContainerStarted","Data":"40d72a8fba8523b57db89a707e928272ed824bf2950d4a14fb455155fb862557"}
May 27 23:35:38.255376 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.255337    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"]
May 27 23:35:38.258838 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.258814    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.261616 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.261564    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 27 23:35:38.262808 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.262656    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 27 23:35:38.262808 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.262670    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 27 23:35:38.262808 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.262709    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-knl2t\""
May 27 23:35:38.262808 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.262717    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 27 23:35:38.262808 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.262737    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 27 23:35:38.269408 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.269367    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"]
May 27 23:35:38.329376 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.329344    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd" event={"ID":"dfee3cde-baf8-4bb9-9a6c-992c1b43e77a","Type":"ContainerStarted","Data":"2e9de559cfdb03d66862668514b7bc3326d905bf9b42a4287422ebeb194a222c"}
May 27 23:35:38.345179 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.345126    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vznvd" podStartSLOduration=2.182668225 podStartE2EDuration="3.345109301s" podCreationTimestamp="2026-05-27 23:35:35 +0000 UTC" firstStartedPulling="2026-05-27 23:35:36.266121312 +0000 UTC m=+126.923983765" lastFinishedPulling="2026-05-27 23:35:37.428562378 +0000 UTC m=+128.086424841" observedRunningTime="2026-05-27 23:35:38.3444956 +0000 UTC m=+129.002358106" watchObservedRunningTime="2026-05-27 23:35:38.345109301 +0000 UTC m=+129.002971776"
May 27 23:35:38.393567 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.393502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9dde2a-6d83-42f6-9499-c6735905726d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.393567 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.393558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce9dde2a-6d83-42f6-9499-c6735905726d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.393849 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.393726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbmc\" (UniqueName: \"kubernetes.io/projected/ce9dde2a-6d83-42f6-9499-c6735905726d-kube-api-access-9hbmc\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.393849 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.393767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce9dde2a-6d83-42f6-9499-c6735905726d-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.494538 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.494483    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9dde2a-6d83-42f6-9499-c6735905726d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.494722 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.494570    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce9dde2a-6d83-42f6-9499-c6735905726d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.494722 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:38.494611    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 27 23:35:38.494722 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:38.494683    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9dde2a-6d83-42f6-9499-c6735905726d-prometheus-operator-tls podName:ce9dde2a-6d83-42f6-9499-c6735905726d nodeName:}" failed. No retries permitted until 2026-05-27 23:35:38.994662934 +0000 UTC m=+129.652525400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ce9dde2a-6d83-42f6-9499-c6735905726d-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-j2sjf" (UID: "ce9dde2a-6d83-42f6-9499-c6735905726d") : secret "prometheus-operator-tls" not found
May 27 23:35:38.494888 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.494754    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbmc\" (UniqueName: \"kubernetes.io/projected/ce9dde2a-6d83-42f6-9499-c6735905726d-kube-api-access-9hbmc\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.494888 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.494792    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce9dde2a-6d83-42f6-9499-c6735905726d-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.495427 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.495401    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce9dde2a-6d83-42f6-9499-c6735905726d-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.497276 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.497245    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce9dde2a-6d83-42f6-9499-c6735905726d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.510101 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.510030    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbmc\" (UniqueName: \"kubernetes.io/projected/ce9dde2a-6d83-42f6-9499-c6735905726d-kube-api-access-9hbmc\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:38.999382 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:38.999335    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9dde2a-6d83-42f6-9499-c6735905726d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:39.002235 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:39.002204    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9dde2a-6d83-42f6-9499-c6735905726d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-j2sjf\" (UID: \"ce9dde2a-6d83-42f6-9499-c6735905726d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:39.170012 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:39.169968    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"
May 27 23:35:39.707010 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:39.706970    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:35:39.710305 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:39.710216    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4abd5dc-33fe-4e84-a433-06f2916dab75-metrics-certs\") pod \"network-metrics-daemon-jd94m\" (UID: \"c4abd5dc-33fe-4e84-a433-06f2916dab75\") " pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:35:39.751762 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:39.751730    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf"]
May 27 23:35:39.755483 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:39.755455    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9dde2a_6d83_42f6_9499_c6735905726d.slice/crio-310b27b050998db912a7f8b5966076159fc6c2f43eab9ebe5870ed74d151b96e WatchSource:0}: Error finding container 310b27b050998db912a7f8b5966076159fc6c2f43eab9ebe5870ed74d151b96e: Status 404 returned error can't find the container with id 310b27b050998db912a7f8b5966076159fc6c2f43eab9ebe5870ed74d151b96e
May 27 23:35:39.761055 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:39.761033    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8jlf4\""
May 27 23:35:39.768998 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:39.768972    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd94m"
May 27 23:35:39.893612 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:39.893520    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jd94m"]
May 27 23:35:39.896290 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:39.896267    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4abd5dc_33fe_4e84_a433_06f2916dab75.slice/crio-f1532828b090c0d003f71e06d4912b6ff0952d66af7993f94ba8af322553341c WatchSource:0}: Error finding container f1532828b090c0d003f71e06d4912b6ff0952d66af7993f94ba8af322553341c: Status 404 returned error can't find the container with id f1532828b090c0d003f71e06d4912b6ff0952d66af7993f94ba8af322553341c
May 27 23:35:40.337561 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:40.337500    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvmmg" event={"ID":"feb33324-55b0-422e-85f3-2ad64b45c12d","Type":"ContainerStarted","Data":"7bb92cbf51693aa36e61fbadbd4fcfad6b297d35a3cbcb40122a678ec4685258"}
May 27 23:35:40.339173 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:40.339126    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf" event={"ID":"ce9dde2a-6d83-42f6-9499-c6735905726d","Type":"ContainerStarted","Data":"310b27b050998db912a7f8b5966076159fc6c2f43eab9ebe5870ed74d151b96e"}
May 27 23:35:40.340699 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:40.340668    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdf79cb5c-hwdg9" event={"ID":"ac05b693-5665-41ba-8647-9b49ad5e7ebe","Type":"ContainerStarted","Data":"6c7227543dad8def517c0fb5e8c7638fff3cb4fff953409ac88f1b2b90c4bdb2"}
May 27 23:35:40.341935 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:40.341905    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jd94m" event={"ID":"c4abd5dc-33fe-4e84-a433-06f2916dab75","Type":"ContainerStarted","Data":"f1532828b090c0d003f71e06d4912b6ff0952d66af7993f94ba8af322553341c"}
May 27 23:35:40.360254 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:40.360189    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wvmmg" podStartSLOduration=2.195380759 podStartE2EDuration="5.36016961s" podCreationTimestamp="2026-05-27 23:35:35 +0000 UTC" firstStartedPulling="2026-05-27 23:35:36.449784759 +0000 UTC m=+127.107647212" lastFinishedPulling="2026-05-27 23:35:39.614573607 +0000 UTC m=+130.272436063" observedRunningTime="2026-05-27 23:35:40.358643155 +0000 UTC m=+131.016505631" watchObservedRunningTime="2026-05-27 23:35:40.36016961 +0000 UTC m=+131.018032086"
May 27 23:35:40.376315 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:40.376260    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6fdf79cb5c-hwdg9" podStartSLOduration=2.140551478 podStartE2EDuration="5.376244152s" podCreationTimestamp="2026-05-27 23:35:35 +0000 UTC" firstStartedPulling="2026-05-27 23:35:36.378870532 +0000 UTC m=+127.036732997" lastFinishedPulling="2026-05-27 23:35:39.614563207 +0000 UTC m=+130.272425671" observedRunningTime="2026-05-27 23:35:40.374978327 +0000 UTC m=+131.032840802" watchObservedRunningTime="2026-05-27 23:35:40.376244152 +0000 UTC m=+131.034106626"
May 27 23:35:41.346391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:41.346345    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf" event={"ID":"ce9dde2a-6d83-42f6-9499-c6735905726d","Type":"ContainerStarted","Data":"9835c6efa378bdab88e21f54d4e183c4783e81ac3ad142e0c2e6d34555f8b50b"}
May 27 23:35:41.346391 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:41.346383    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf" event={"ID":"ce9dde2a-6d83-42f6-9499-c6735905726d","Type":"ContainerStarted","Data":"2b55e50ddde4e75699035326bf134213dd93d082c5e468242fa4f729945ffb17"}
May 27 23:35:41.347831 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:41.347805    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jd94m" event={"ID":"c4abd5dc-33fe-4e84-a433-06f2916dab75","Type":"ContainerStarted","Data":"72ad900e2f990ead4cfd6ab3199f3d3c9f69e8cd11867ace7627973f2fc23503"}
May 27 23:35:41.366972 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:41.366911    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-j2sjf" podStartSLOduration=1.9442614900000001 podStartE2EDuration="3.366889901s" podCreationTimestamp="2026-05-27 23:35:38 +0000 UTC" firstStartedPulling="2026-05-27 23:35:39.757368861 +0000 UTC m=+130.415231316" lastFinishedPulling="2026-05-27 23:35:41.179997275 +0000 UTC m=+131.837859727" observedRunningTime="2026-05-27 23:35:41.364298464 +0000 UTC m=+132.022161246" watchObservedRunningTime="2026-05-27 23:35:41.366889901 +0000 UTC m=+132.024752378"
May 27 23:35:42.352036 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:42.351994    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jd94m" event={"ID":"c4abd5dc-33fe-4e84-a433-06f2916dab75","Type":"ContainerStarted","Data":"4b6fe4ddd2bae6ef073c6e16ac756a51d65de185c98a046a0bf0f4cd67104ef9"}
May 27 23:35:42.371877 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:42.369701    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jd94m" podStartSLOduration=131.085321358 podStartE2EDuration="2m12.369682667s" podCreationTimestamp="2026-05-27 23:33:30 +0000 UTC" firstStartedPulling="2026-05-27 23:35:39.898083929 +0000 UTC m=+130.555946384" lastFinishedPulling="2026-05-27 23:35:41.182445236 +0000 UTC m=+131.840307693" observedRunningTime="2026-05-27 23:35:42.368243715 +0000 UTC m=+133.026106189" watchObservedRunningTime="2026-05-27 23:35:42.369682667 +0000 UTC m=+133.027545141"
May 27 23:35:43.626904 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.626870    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"]
May 27 23:35:43.630799 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.630777    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.633814 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.633782    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 27 23:35:43.633814 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.633797    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 27 23:35:43.634202 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.633947    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l25d8\""
May 27 23:35:43.637176 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.637151    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kqdhj"]
May 27 23:35:43.640678 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.640652    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.641211 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.641189    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"]
May 27 23:35:43.643505 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.643481    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 27 23:35:43.643902 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.643878    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 27 23:35:43.644150 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.644134    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pbz7x\""
May 27 23:35:43.644456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.644436    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 27 23:35:43.743606 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f661370-05a6-49dc-a2db-0e20de19d746-sys\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.743606 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743610    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a40181-7f04-4107-862b-65fb6861574c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.743810 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743633    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z495\" (UniqueName: \"kubernetes.io/projected/6f661370-05a6-49dc-a2db-0e20de19d746-kube-api-access-5z495\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.743810 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jtw\" (UniqueName: \"kubernetes.io/projected/96a40181-7f04-4107-862b-65fb6861574c-kube-api-access-25jtw\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.743810 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f661370-05a6-49dc-a2db-0e20de19d746-root\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.743810 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-tls\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.743810 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-wtmp\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.743810 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f661370-05a6-49dc-a2db-0e20de19d746-metrics-client-ca\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.743991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743787    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96a40181-7f04-4107-862b-65fb6861574c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.743991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96a40181-7f04-4107-862b-65fb6861574c-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.743991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-accelerators-collector-config\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.743991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-textfile\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.743991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.743938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.844494 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844452    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96a40181-7f04-4107-862b-65fb6861574c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.844494 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844492    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96a40181-7f04-4107-862b-65fb6861574c-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.844707 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844621    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-accelerators-collector-config\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.844707 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844682    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-textfile\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.844822 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844710    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.844822 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844748    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f661370-05a6-49dc-a2db-0e20de19d746-sys\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.844822 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844784    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a40181-7f04-4107-862b-65fb6861574c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.844956 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844866    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f661370-05a6-49dc-a2db-0e20de19d746-sys\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.844956 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:43.844875    2562 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 27 23:35:43.844956 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844911    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5z495\" (UniqueName: \"kubernetes.io/projected/6f661370-05a6-49dc-a2db-0e20de19d746-kube-api-access-5z495\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.845153 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.844957    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-25jtw\" (UniqueName: \"kubernetes.io/projected/96a40181-7f04-4107-862b-65fb6861574c-kube-api-access-25jtw\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.845153 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:43.844980    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a40181-7f04-4107-862b-65fb6861574c-openshift-state-metrics-tls podName:96a40181-7f04-4107-862b-65fb6861574c nodeName:}" failed. No retries permitted until 2026-05-27 23:35:44.344958271 +0000 UTC m=+135.002820733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/96a40181-7f04-4107-862b-65fb6861574c-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-zb5sf" (UID: "96a40181-7f04-4107-862b-65fb6861574c") : secret "openshift-state-metrics-tls" not found
May 27 23:35:43.845153 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845023    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f661370-05a6-49dc-a2db-0e20de19d746-root\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.845153 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845052    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-tls\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.845153 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845081    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-wtmp\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.845153 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845122    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f661370-05a6-49dc-a2db-0e20de19d746-metrics-client-ca\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.845438 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845225    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96a40181-7f04-4107-862b-65fb6861574c-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.845438 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845241    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-accelerators-collector-config\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.845438 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845288    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f661370-05a6-49dc-a2db-0e20de19d746-root\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.845438 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845301    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-wtmp\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.845648 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845571    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-textfile\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.845765 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.845744    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f661370-05a6-49dc-a2db-0e20de19d746-metrics-client-ca\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.847017 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.846999    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96a40181-7f04-4107-862b-65fb6861574c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.847590 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.847572    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.847845 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.847830    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f661370-05a6-49dc-a2db-0e20de19d746-node-exporter-tls\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.855944 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.855914    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z495\" (UniqueName: \"kubernetes.io/projected/6f661370-05a6-49dc-a2db-0e20de19d746-kube-api-access-5z495\") pod \"node-exporter-kqdhj\" (UID: \"6f661370-05a6-49dc-a2db-0e20de19d746\") " pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.856077 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.855981    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jtw\" (UniqueName: \"kubernetes.io/projected/96a40181-7f04-4107-862b-65fb6861574c-kube-api-access-25jtw\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:43.949583 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:43.949471    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kqdhj"
May 27 23:35:43.957263 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:43.957234    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f661370_05a6_49dc_a2db_0e20de19d746.slice/crio-8d9e6c62078375662043c7c73d10648e693d0057bfc6262fad516035cae463dd WatchSource:0}: Error finding container 8d9e6c62078375662043c7c73d10648e693d0057bfc6262fad516035cae463dd: Status 404 returned error can't find the container with id 8d9e6c62078375662043c7c73d10648e693d0057bfc6262fad516035cae463dd
May 27 23:35:44.350184 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.350140    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a40181-7f04-4107-862b-65fb6861574c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:44.353034 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.353007    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a40181-7f04-4107-862b-65fb6861574c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-zb5sf\" (UID: \"96a40181-7f04-4107-862b-65fb6861574c\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:44.358593 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.358556    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kqdhj" event={"ID":"6f661370-05a6-49dc-a2db-0e20de19d746","Type":"ContainerStarted","Data":"8d9e6c62078375662043c7c73d10648e693d0057bfc6262fad516035cae463dd"}
May 27 23:35:44.540501 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.540467    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"
May 27 23:35:44.684354 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.684324    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf"]
May 27 23:35:44.696603 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.696572    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 27 23:35:44.701506 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.701481    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.704096 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704072    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 27 23:35:44.704212 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704110    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 27 23:35:44.704212 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704125    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 27 23:35:44.704212 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704153    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5dk6l\""
May 27 23:35:44.704439 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704423    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 27 23:35:44.704490 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704440    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 27 23:35:44.704490 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704454    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 27 23:35:44.704490 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704471    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 27 23:35:44.704661 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704441    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 27 23:35:44.704908 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.704887    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 27 23:35:44.717176 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.717131    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 27 23:35:44.816401 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:44.816365    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a40181_7f04_4107_862b_65fb6861574c.slice/crio-df8957ff8f918ba85a9279506d1d8b24a46ea02fad6c61e8c387ff669f8f3c49 WatchSource:0}: Error finding container df8957ff8f918ba85a9279506d1d8b24a46ea02fad6c61e8c387ff669f8f3c49: Status 404 returned error can't find the container with id df8957ff8f918ba85a9279506d1d8b24a46ea02fad6c61e8c387ff669f8f3c49
May 27 23:35:44.854579 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-config-out\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.854701 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.854701 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854629    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.854701 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.854907 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.854907 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.854907 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.854907 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854862    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.854907 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdzp2\" (UniqueName: \"kubernetes.io/projected/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-kube-api-access-kdzp2\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.855160 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.855160 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-config-volume\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.855160 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.854978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.855160 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.855012    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-web-config\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.955936 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.955905    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956058 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.955954    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kdzp2\" (UniqueName: \"kubernetes.io/projected/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-kube-api-access-kdzp2\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956058 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.955998    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956175 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.956061    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-config-volume\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956175 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.956106    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956508 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.956483    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-web-config\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956612 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.956559    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-config-out\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956612 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.956595    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956726 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.956623    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956726 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.956692    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956726 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.956717    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.956876 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:44.956807    2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 27 23:35:44.956876 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:35:44.956864    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-main-tls podName:3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00 nodeName:}" failed. No retries permitted until 2026-05-27 23:35:45.456843493 +0000 UTC m=+136.114705962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00") : secret "alertmanager-main-tls" not found
May 27 23:35:44.957504 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.956725    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.957504 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.957070    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.957504 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.957102    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.958324 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.958288    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.958420 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.958288    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.959255 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.959234    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-config-out\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.959783 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.959751    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.959870 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.959816    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-web-config\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.960359 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.960292    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-config-volume\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.960760 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.960728    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.960834 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.960780    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.960975 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.960929    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.962179 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.962153    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:44.966900 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:44.966879    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdzp2\" (UniqueName: \"kubernetes.io/projected/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-kube-api-access-kdzp2\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:45.363755 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:45.363721    2562 generic.go:358] "Generic (PLEG): container finished" podID="6f661370-05a6-49dc-a2db-0e20de19d746" containerID="4e7f64776e3937d40a3c87f656f47f87e5d95881d439e84374d5c85a64b24b4a" exitCode=0
May 27 23:35:45.363954 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:45.363813    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kqdhj" event={"ID":"6f661370-05a6-49dc-a2db-0e20de19d746","Type":"ContainerDied","Data":"4e7f64776e3937d40a3c87f656f47f87e5d95881d439e84374d5c85a64b24b4a"}
May 27 23:35:45.365793 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:45.365765    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf" event={"ID":"96a40181-7f04-4107-862b-65fb6861574c","Type":"ContainerStarted","Data":"3c367b95c804140062fb1d7d48eea72a70fc50f8b9cb75358e9d758988cb62d3"}
May 27 23:35:45.365920 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:45.365803    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf" event={"ID":"96a40181-7f04-4107-862b-65fb6861574c","Type":"ContainerStarted","Data":"d3587d3b95d1b92bae2cc083e506b73a118f05c0b973749002dc9f23be3b23d7"}
May 27 23:35:45.365920 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:45.365818    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf" event={"ID":"96a40181-7f04-4107-862b-65fb6861574c","Type":"ContainerStarted","Data":"df8957ff8f918ba85a9279506d1d8b24a46ea02fad6c61e8c387ff669f8f3c49"}
May 27 23:35:45.461666 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:45.461622    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:45.464330 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:45.464303    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00\") " pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:45.612102 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:45.612068    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 27 23:35:45.861562 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:45.861511    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 27 23:35:45.862260 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:45.862231    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b21d1a6_e11a_4e2f_b65f_44f1a5bb7d00.slice/crio-62c73a66e672ef659edc53143789b546240200fa76d984dc5a41c0e14fa3b08a WatchSource:0}: Error finding container 62c73a66e672ef659edc53143789b546240200fa76d984dc5a41c0e14fa3b08a: Status 404 returned error can't find the container with id 62c73a66e672ef659edc53143789b546240200fa76d984dc5a41c0e14fa3b08a
May 27 23:35:46.241809 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.241767    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:46.241809 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.241818    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:35:46.243065 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.243039    2562 patch_prober.go:28] interesting pod/console-6fdf79cb5c-hwdg9 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused" start-of-body=
May 27 23:35:46.243154 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.243084    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6fdf79cb5c-hwdg9" podUID="ac05b693-5665-41ba-8647-9b49ad5e7ebe" containerName="console" probeResult="failure" output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused"
May 27 23:35:46.371136 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.371100    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kqdhj" event={"ID":"6f661370-05a6-49dc-a2db-0e20de19d746","Type":"ContainerStarted","Data":"f835ee410d90190f24c55ae9eba0c0da6e552a19268988bfd4b998bff80db450"}
May 27 23:35:46.371136 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.371143    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kqdhj" event={"ID":"6f661370-05a6-49dc-a2db-0e20de19d746","Type":"ContainerStarted","Data":"3af05475cc12b487202422990dcf9961ee13dc7c5bcfa6b65f481da704fb8d41"}
May 27 23:35:46.372245 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.372215    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00","Type":"ContainerStarted","Data":"62c73a66e672ef659edc53143789b546240200fa76d984dc5a41c0e14fa3b08a"}
May 27 23:35:46.373777 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.373747    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf" event={"ID":"96a40181-7f04-4107-862b-65fb6861574c","Type":"ContainerStarted","Data":"04a19ec0c5ef48fa8b77220b42038984aff02a4cefc37ea4e072f67a7421e0e7"}
May 27 23:35:46.390393 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.390350    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kqdhj" podStartSLOduration=2.487452146 podStartE2EDuration="3.390335124s" podCreationTimestamp="2026-05-27 23:35:43 +0000 UTC" firstStartedPulling="2026-05-27 23:35:43.95885201 +0000 UTC m=+134.616714462" lastFinishedPulling="2026-05-27 23:35:44.861734987 +0000 UTC m=+135.519597440" observedRunningTime="2026-05-27 23:35:46.389363294 +0000 UTC m=+137.047225770" watchObservedRunningTime="2026-05-27 23:35:46.390335124 +0000 UTC m=+137.048197598"
May 27 23:35:46.407469 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.407406    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-zb5sf" podStartSLOduration=2.593432217 podStartE2EDuration="3.407390704s" podCreationTimestamp="2026-05-27 23:35:43 +0000 UTC" firstStartedPulling="2026-05-27 23:35:44.961128042 +0000 UTC m=+135.618990494" lastFinishedPulling="2026-05-27 23:35:45.775086512 +0000 UTC m=+136.432948981" observedRunningTime="2026-05-27 23:35:46.40599222 +0000 UTC m=+137.063854695" watchObservedRunningTime="2026-05-27 23:35:46.407390704 +0000 UTC m=+137.065253177"
May 27 23:35:46.726655 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.726565    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"]
May 27 23:35:46.730899 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.730869    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.733899 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.733603    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 27 23:35:46.733899 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.733628    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l48mn\""
May 27 23:35:46.733899 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.733628    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4f29ti0icni3o\""
May 27 23:35:46.733899 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.733658    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 27 23:35:46.733899 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.733737    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 27 23:35:46.733899 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.733786    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 27 23:35:46.733899 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.733804    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 27 23:35:46.738811 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.738781    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"]
May 27 23:35:46.876418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.876385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksnrn\" (UniqueName: \"kubernetes.io/projected/041d9cf1-009b-46e4-a882-d832912ded65-kube-api-access-ksnrn\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.876819 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.876452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.876819 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.876502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-grpc-tls\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.876819 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.876522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.876819 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.876562    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/041d9cf1-009b-46e4-a882-d832912ded65-metrics-client-ca\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.876819 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.876585    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.876819 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.876608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-tls\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.876819 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.876624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.978095 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.978025    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-grpc-tls\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.978095 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.978080    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.978262 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.978138    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/041d9cf1-009b-46e4-a882-d832912ded65-metrics-client-ca\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.978262 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.978192    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.978262 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.978235    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-tls\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.978425 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.978264    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.978425 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.978333    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ksnrn\" (UniqueName: \"kubernetes.io/projected/041d9cf1-009b-46e4-a882-d832912ded65-kube-api-access-ksnrn\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.978566 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.978422    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.979141 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.979046    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/041d9cf1-009b-46e4-a882-d832912ded65-metrics-client-ca\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.980945 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.980902    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.981137 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.981104    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.981260 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.981233    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-grpc-tls\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.981361 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.981302    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.981493 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.981472    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-tls\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.981680 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.981659    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/041d9cf1-009b-46e4-a882-d832912ded65-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:46.986687 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:46.986666    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksnrn\" (UniqueName: \"kubernetes.io/projected/041d9cf1-009b-46e4-a882-d832912ded65-kube-api-access-ksnrn\") pod \"thanos-querier-776d4c97f4-hqmvm\" (UID: \"041d9cf1-009b-46e4-a882-d832912ded65\") " pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:47.042591 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:47.042565    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:47.173109 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:47.169905    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"]
May 27 23:35:47.377387 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:47.377350    2562 generic.go:358] "Generic (PLEG): container finished" podID="3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00" containerID="3d49ff72e2c0639bd65be1d133077a353e6c2d7ab5d315aa0fd7b314491ab9a7" exitCode=0
May 27 23:35:47.377586 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:47.377433    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00","Type":"ContainerDied","Data":"3d49ff72e2c0639bd65be1d133077a353e6c2d7ab5d315aa0fd7b314491ab9a7"}
May 27 23:35:47.378589 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:47.378556    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm" event={"ID":"041d9cf1-009b-46e4-a882-d832912ded65","Type":"ContainerStarted","Data":"846ee0e80e5034743a0bc062608afc09fe5afc75ec4e7ab05fd2aaaa3c913331"}
May 27 23:35:48.225223 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.225138    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"]
May 27 23:35:48.229450 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.229422    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.232798 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.232605    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 27 23:35:48.237504 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.234219    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dk10o40q2ik7a\""
May 27 23:35:48.237504 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.234457    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 27 23:35:48.237504 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.234750    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 27 23:35:48.237504 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.234961    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nxp8b\""
May 27 23:35:48.237504 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.235219    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 27 23:35:48.238369 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.238310    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"]
May 27 23:35:48.390270 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.390230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b19d15-d5cc-41fd-a8fe-26de50041342-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.390470 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.390355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33b19d15-d5cc-41fd-a8fe-26de50041342-client-ca-bundle\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.390470 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.390388    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33b19d15-d5cc-41fd-a8fe-26de50041342-audit-log\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.390470 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.390421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hr4w\" (UniqueName: \"kubernetes.io/projected/33b19d15-d5cc-41fd-a8fe-26de50041342-kube-api-access-7hr4w\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.390470 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.390467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33b19d15-d5cc-41fd-a8fe-26de50041342-secret-metrics-server-tls\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.390731 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.390489    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33b19d15-d5cc-41fd-a8fe-26de50041342-metrics-server-audit-profiles\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.390731 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.390509    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33b19d15-d5cc-41fd-a8fe-26de50041342-secret-metrics-server-client-certs\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.405872 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.405836    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"]
May 27 23:35:48.411101 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.410741    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"
May 27 23:35:48.413645 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.413619    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 27 23:35:48.414155 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.413833    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-78sk8\""
May 27 23:35:48.421493 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.421267    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"]
May 27 23:35:48.492069 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.492026    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b19d15-d5cc-41fd-a8fe-26de50041342-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.492252 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.492135    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33b19d15-d5cc-41fd-a8fe-26de50041342-client-ca-bundle\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.492252 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.492166    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33b19d15-d5cc-41fd-a8fe-26de50041342-audit-log\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.492252 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.492201    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7hr4w\" (UniqueName: \"kubernetes.io/projected/33b19d15-d5cc-41fd-a8fe-26de50041342-kube-api-access-7hr4w\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.492252 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.492248    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33b19d15-d5cc-41fd-a8fe-26de50041342-secret-metrics-server-tls\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.492456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.492273    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33b19d15-d5cc-41fd-a8fe-26de50041342-metrics-server-audit-profiles\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.492456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.492304    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33b19d15-d5cc-41fd-a8fe-26de50041342-secret-metrics-server-client-certs\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.493063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.493034    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33b19d15-d5cc-41fd-a8fe-26de50041342-audit-log\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.493696 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.493647    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b19d15-d5cc-41fd-a8fe-26de50041342-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.493876 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.493823    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33b19d15-d5cc-41fd-a8fe-26de50041342-metrics-server-audit-profiles\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.495406 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.495381    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33b19d15-d5cc-41fd-a8fe-26de50041342-secret-metrics-server-client-certs\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.496454 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.496171    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33b19d15-d5cc-41fd-a8fe-26de50041342-secret-metrics-server-tls\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.496454 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.496251    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33b19d15-d5cc-41fd-a8fe-26de50041342-client-ca-bundle\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.502109 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.502086    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hr4w\" (UniqueName: \"kubernetes.io/projected/33b19d15-d5cc-41fd-a8fe-26de50041342-kube-api-access-7hr4w\") pod \"metrics-server-5f94fbfbcf-hpgxp\" (UID: \"33b19d15-d5cc-41fd-a8fe-26de50041342\") " pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.546698 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.546657    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:35:48.593201 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.593167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f104bebb-24e5-4530-8f45-a68563422250-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-46x9j\" (UID: \"f104bebb-24e5-4530-8f45-a68563422250\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"
May 27 23:35:48.694108 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.694078    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f104bebb-24e5-4530-8f45-a68563422250-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-46x9j\" (UID: \"f104bebb-24e5-4530-8f45-a68563422250\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"
May 27 23:35:48.696548 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.696499    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f104bebb-24e5-4530-8f45-a68563422250-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-46x9j\" (UID: \"f104bebb-24e5-4530-8f45-a68563422250\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"
May 27 23:35:48.724428 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.724393    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"
May 27 23:35:48.884412 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.884317    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8f56489d8-rkmqn"]
May 27 23:35:48.888523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.888493    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.891803 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.891584    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-crkcg\""
May 27 23:35:48.891803 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.891584    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 27 23:35:48.891803 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.891585    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 27 23:35:48.892242 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.892221    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 27 23:35:48.892351 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.892261    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 27 23:35:48.892351 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.892267    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 27 23:35:48.895427 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.895397    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-serving-certs-ca-bundle\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.895554 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.895437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-metrics-client-ca\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.895554 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.895454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdmt\" (UniqueName: \"kubernetes.io/projected/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-kube-api-access-9xdmt\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.895693 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.895564    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-federate-client-tls\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.895693 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.895632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-telemeter-client-tls\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.895693 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.895676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.895806 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.895751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.895806 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.895781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-secret-telemeter-client\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.899777 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.899734    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8f56489d8-rkmqn"]
May 27 23:35:48.904188 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.904165    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 27 23:35:48.996749 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.996711    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-federate-client-tls\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.996939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.996764    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-telemeter-client-tls\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.996939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.996797    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.996939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.996847    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.996939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.996875    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-secret-telemeter-client\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.996939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.996938    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-serving-certs-ca-bundle\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.997183 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.996969    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-metrics-client-ca\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.997183 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.996993    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdmt\" (UniqueName: \"kubernetes.io/projected/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-kube-api-access-9xdmt\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.997722 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.997695    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-serving-certs-ca-bundle\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.997904 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.997879    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.998122 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.998100    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-metrics-client-ca\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.999564 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.999520    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-federate-client-tls\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.999692 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.999670    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:48.999926 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.999907    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-telemeter-client-tls\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:49.000022 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:48.999994    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-secret-telemeter-client\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:49.006501 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:49.006475    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdmt\" (UniqueName: \"kubernetes.io/projected/8a2ae9b1-9e23-4259-bcc0-72b1c8305010-kube-api-access-9xdmt\") pod \"telemeter-client-8f56489d8-rkmqn\" (UID: \"8a2ae9b1-9e23-4259-bcc0-72b1c8305010\") " pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:49.201061 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:49.200986    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn"
May 27 23:35:49.392405 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:49.392135    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm" event={"ID":"041d9cf1-009b-46e4-a882-d832912ded65","Type":"ContainerStarted","Data":"57986c0fe7491351f0dbf913f4e452a176ee2676f353d4ea8acbb21c593d2aa2"}
May 27 23:35:49.395265 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:49.395240    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00","Type":"ContainerStarted","Data":"c37a57929fc6e50a0ad6aa0d7e518e54b1d9c8d7c45dcbbdeb42fcc317f6ab0e"}
May 27 23:35:49.410718 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:49.410609    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"]
May 27 23:35:49.419936 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:49.419892    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf104bebb_24e5_4530_8f45_a68563422250.slice/crio-d613c5e1a1a731e0dad34748a49610587726b0440786afb300d34ecfde74a64e WatchSource:0}: Error finding container d613c5e1a1a731e0dad34748a49610587726b0440786afb300d34ecfde74a64e: Status 404 returned error can't find the container with id d613c5e1a1a731e0dad34748a49610587726b0440786afb300d34ecfde74a64e
May 27 23:35:49.447031 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:49.446980    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8f56489d8-rkmqn"]
May 27 23:35:49.449828 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:49.449781    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2ae9b1_9e23_4259_bcc0_72b1c8305010.slice/crio-0ca3288c064de4e4de3b25dae026a9cd9c56556747887eaecf42f64d4bd1d0f3 WatchSource:0}: Error finding container 0ca3288c064de4e4de3b25dae026a9cd9c56556747887eaecf42f64d4bd1d0f3: Status 404 returned error can't find the container with id 0ca3288c064de4e4de3b25dae026a9cd9c56556747887eaecf42f64d4bd1d0f3
May 27 23:35:49.468741 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:49.468714    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"]
May 27 23:35:49.473166 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:49.473142    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b19d15_d5cc_41fd_a8fe_26de50041342.slice/crio-78d7033a9051f0ff1320d2a383a58b7621ebadb06396e11c7953bf23b99b520b WatchSource:0}: Error finding container 78d7033a9051f0ff1320d2a383a58b7621ebadb06396e11c7953bf23b99b520b: Status 404 returned error can't find the container with id 78d7033a9051f0ff1320d2a383a58b7621ebadb06396e11c7953bf23b99b520b
May 27 23:35:50.403086 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:50.403012    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm" event={"ID":"041d9cf1-009b-46e4-a882-d832912ded65","Type":"ContainerStarted","Data":"9ab7a236346b457146909863f59e6a713114bd59f59e2d7026ce18537ae92cf0"}
May 27 23:35:50.403086 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:50.403057    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm" event={"ID":"041d9cf1-009b-46e4-a882-d832912ded65","Type":"ContainerStarted","Data":"2f1a6a2d88fcae52ed4430b47886ff418ffff0d118313802bc494df7c7e21b2c"}
May 27 23:35:50.406156 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:50.405938    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn" event={"ID":"8a2ae9b1-9e23-4259-bcc0-72b1c8305010","Type":"ContainerStarted","Data":"0ca3288c064de4e4de3b25dae026a9cd9c56556747887eaecf42f64d4bd1d0f3"}
May 27 23:35:50.409842 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:50.409812    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00","Type":"ContainerStarted","Data":"f581c4577106890ac245087d6a24ebabe120ae9fce97c92638b9d40432728a20"}
May 27 23:35:50.409978 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:50.409878    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00","Type":"ContainerStarted","Data":"3269df05f7334c9a8cfbf66649caa966b9a61c368a4acf0ff05a578dd15b119e"}
May 27 23:35:50.409978 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:50.409897    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00","Type":"ContainerStarted","Data":"c15d7e9ed1ccfc356c8798710384e0249a1f457bfa4522c5f7b0f65aafcf8dcc"}
May 27 23:35:50.409978 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:50.409908    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00","Type":"ContainerStarted","Data":"84d1b33dc26896fecf674a5a14b72bc2d657d1e5ae179fb48acd625366412b69"}
May 27 23:35:50.411403 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:50.411299    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j" event={"ID":"f104bebb-24e5-4530-8f45-a68563422250","Type":"ContainerStarted","Data":"d613c5e1a1a731e0dad34748a49610587726b0440786afb300d34ecfde74a64e"}
May 27 23:35:50.414112 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:50.414072    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp" event={"ID":"33b19d15-d5cc-41fd-a8fe-26de50041342","Type":"ContainerStarted","Data":"78d7033a9051f0ff1320d2a383a58b7621ebadb06396e11c7953bf23b99b520b"}
May 27 23:35:52.319302 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.319266    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fdf79cb5c-hwdg9"]
May 27 23:35:52.357293 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.357196    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-cf9596469-b2s4d"]
May 27 23:35:52.360544 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.360510    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.368257 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.368234    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 27 23:35:52.371112 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.371085    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cf9596469-b2s4d"]
May 27 23:35:52.423185 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.423148    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm" event={"ID":"041d9cf1-009b-46e4-a882-d832912ded65","Type":"ContainerStarted","Data":"c84d324b6a77158fdbf5acaab5faaf9937cd57b7abda71e3e061e0cb4a6afef4"}
May 27 23:35:52.423185 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.423189    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm" event={"ID":"041d9cf1-009b-46e4-a882-d832912ded65","Type":"ContainerStarted","Data":"8f3573d556ca493cbb0aef53234f04443e45749f73bd52f0c518d8199b7b3ccd"}
May 27 23:35:52.423434 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.423202    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm" event={"ID":"041d9cf1-009b-46e4-a882-d832912ded65","Type":"ContainerStarted","Data":"42ee4bc332bb860fae1696845e1f7a3ce374c3e607b68211e8d1664268b63891"}
May 27 23:35:52.423434 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.423317    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:35:52.425051 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.425024    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn" event={"ID":"8a2ae9b1-9e23-4259-bcc0-72b1c8305010","Type":"ContainerStarted","Data":"803b7fc41b00243816e4c122aa9de5c64f9e3d6fc0246426b20537d89e55f3d9"}
May 27 23:35:52.425163 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.425056    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn" event={"ID":"8a2ae9b1-9e23-4259-bcc0-72b1c8305010","Type":"ContainerStarted","Data":"20e43bee514dbcfcd62cc419cee07d49fc3ac60905cb42e71630da0eb8e2f6f4"}
May 27 23:35:52.425163 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.425067    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn" event={"ID":"8a2ae9b1-9e23-4259-bcc0-72b1c8305010","Type":"ContainerStarted","Data":"df64e21f9f61044bb6817bf6f58d34b9bdeca4886d4dbf7a5636e950fb06464d"}
May 27 23:35:52.428250 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.428223    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00","Type":"ContainerStarted","Data":"7d2a4e89db0b879a0f043805ad4188edba2a39007d871d89e73bf11b49dc82e0"}
May 27 23:35:52.429623 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.429602    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j" event={"ID":"f104bebb-24e5-4530-8f45-a68563422250","Type":"ContainerStarted","Data":"db23019a63e6bb87ba1fb7c6339793cec9c8e60f4f408fc6d22963ae631dbefc"}
May 27 23:35:52.429782 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.429760    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"
May 27 23:35:52.431028 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.431009    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp" event={"ID":"33b19d15-d5cc-41fd-a8fe-26de50041342","Type":"ContainerStarted","Data":"052878758eaa877e1cae17a7674bab49ff218275c76e358cff2d9ed12a52d327"}
May 27 23:35:52.436117 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.436096    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j"
May 27 23:35:52.443680 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.443651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-service-ca\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.443873 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.443852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlgf\" (UniqueName: \"kubernetes.io/projected/eaf47643-d030-4965-9847-13704743ef97-kube-api-access-pvlgf\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.443993 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.443980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-oauth-config\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.444315 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.444253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-trusted-ca-bundle\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.444409 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.444349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-serving-cert\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.444545 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.444448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-oauth-serving-cert\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.444605 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.444591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-console-config\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.446857 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.446814    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm" podStartSLOduration=1.751856702 podStartE2EDuration="6.446802369s" podCreationTimestamp="2026-05-27 23:35:46 +0000 UTC" firstStartedPulling="2026-05-27 23:35:47.175015145 +0000 UTC m=+137.832877596" lastFinishedPulling="2026-05-27 23:35:51.869960797 +0000 UTC m=+142.527823263" observedRunningTime="2026-05-27 23:35:52.445595826 +0000 UTC m=+143.103458297" watchObservedRunningTime="2026-05-27 23:35:52.446802369 +0000 UTC m=+143.104664842"
May 27 23:35:52.475445 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.475394    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.4696365399999998 podStartE2EDuration="8.475377166s" podCreationTimestamp="2026-05-27 23:35:44 +0000 UTC" firstStartedPulling="2026-05-27 23:35:45.86489 +0000 UTC m=+136.522752467" lastFinishedPulling="2026-05-27 23:35:51.870630641 +0000 UTC m=+142.528493093" observedRunningTime="2026-05-27 23:35:52.473323463 +0000 UTC m=+143.131185940" watchObservedRunningTime="2026-05-27 23:35:52.475377166 +0000 UTC m=+143.133239674"
May 27 23:35:52.499783 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.495031    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp" podStartSLOduration=2.096454943 podStartE2EDuration="4.49501182s" podCreationTimestamp="2026-05-27 23:35:48 +0000 UTC" firstStartedPulling="2026-05-27 23:35:49.475309831 +0000 UTC m=+140.133172283" lastFinishedPulling="2026-05-27 23:35:51.873866707 +0000 UTC m=+142.531729160" observedRunningTime="2026-05-27 23:35:52.49098589 +0000 UTC m=+143.148848365" watchObservedRunningTime="2026-05-27 23:35:52.49501182 +0000 UTC m=+143.152874272"
May 27 23:35:52.518595 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.518511    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8f56489d8-rkmqn" podStartSLOduration=2.096950344 podStartE2EDuration="4.518491001s" podCreationTimestamp="2026-05-27 23:35:48 +0000 UTC" firstStartedPulling="2026-05-27 23:35:49.451877495 +0000 UTC m=+140.109739951" lastFinishedPulling="2026-05-27 23:35:51.873418153 +0000 UTC m=+142.531280608" observedRunningTime="2026-05-27 23:35:52.515358077 +0000 UTC m=+143.173220588" watchObservedRunningTime="2026-05-27 23:35:52.518491001 +0000 UTC m=+143.176353476"
May 27 23:35:52.534035 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.533964    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-46x9j" podStartSLOduration=2.086504765 podStartE2EDuration="4.533943348s" podCreationTimestamp="2026-05-27 23:35:48 +0000 UTC" firstStartedPulling="2026-05-27 23:35:49.422692993 +0000 UTC m=+140.080555468" lastFinishedPulling="2026-05-27 23:35:51.8701316 +0000 UTC m=+142.527994051" observedRunningTime="2026-05-27 23:35:52.53378295 +0000 UTC m=+143.191645436" watchObservedRunningTime="2026-05-27 23:35:52.533943348 +0000 UTC m=+143.191805823"
May 27 23:35:52.545473 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.545436    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-console-config\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.545670 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.545588    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-service-ca\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.545670 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.545628    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlgf\" (UniqueName: \"kubernetes.io/projected/eaf47643-d030-4965-9847-13704743ef97-kube-api-access-pvlgf\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.545670 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.545659    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-oauth-config\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.545836 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.545705    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-trusted-ca-bundle\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.545836 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.545736    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-serving-cert\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.545836 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.545761    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-oauth-serving-cert\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.546325 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.546285    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-console-config\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.546413 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.546305    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-service-ca\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.546508 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.546488    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-oauth-serving-cert\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.547108 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.547083    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-trusted-ca-bundle\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.548542 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.548510    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-oauth-config\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.548746 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.548728    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-serving-cert\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.557008 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.556984    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlgf\" (UniqueName: \"kubernetes.io/projected/eaf47643-d030-4965-9847-13704743ef97-kube-api-access-pvlgf\") pod \"console-cf9596469-b2s4d\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") " pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.671764 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.671662    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:35:52.803805 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.803776    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cf9596469-b2s4d"]
May 27 23:35:52.805705 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:52.805677    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf47643_d030_4965_9847_13704743ef97.slice/crio-24799d3c98d5f19b5dbb6f9cd2322decc5ff5851d97b70d4a9d45d07d26ae27f WatchSource:0}: Error finding container 24799d3c98d5f19b5dbb6f9cd2322decc5ff5851d97b70d4a9d45d07d26ae27f: Status 404 returned error can't find the container with id 24799d3c98d5f19b5dbb6f9cd2322decc5ff5851d97b70d4a9d45d07d26ae27f
May 27 23:35:52.957909 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.957834    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c6dcc968c-g4gsk"]
May 27 23:35:52.961236 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.961218    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:52.973106 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:52.973081    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6dcc968c-g4gsk"]
May 27 23:35:53.050077 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.050040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-trusted-ca-bundle\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.050231 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.050108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-oauth-config\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.050231 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.050182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-service-ca\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.050231 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.050225    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-config\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.050367 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.050286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-oauth-serving-cert\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.050367 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.050319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4c4g\" (UniqueName: \"kubernetes.io/projected/e6f9668e-968d-47f5-898d-71fbd3cc64c6-kube-api-access-s4c4g\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.050367 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.050350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-serving-cert\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.151179 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.151126    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-config\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.151384 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.151215    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-oauth-serving-cert\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.151384 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.151263    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c4g\" (UniqueName: \"kubernetes.io/projected/e6f9668e-968d-47f5-898d-71fbd3cc64c6-kube-api-access-s4c4g\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.151384 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.151293    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-serving-cert\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.151384 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.151342    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-trusted-ca-bundle\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.151719 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.151408    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-oauth-config\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.151719 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.151467    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-service-ca\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.152114 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.152063    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-config\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.152114 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.152096    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-oauth-serving-cert\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.152363 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.152334    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-service-ca\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.152760 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.152740    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-trusted-ca-bundle\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.154136 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.154083    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-oauth-config\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.154689 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.154596    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-serving-cert\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.159353 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.159328    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4c4g\" (UniqueName: \"kubernetes.io/projected/e6f9668e-968d-47f5-898d-71fbd3cc64c6-kube-api-access-s4c4g\") pod \"console-6c6dcc968c-g4gsk\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") " pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.271357 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.271331    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:35:53.403180 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.403156    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6dcc968c-g4gsk"]
May 27 23:35:53.405779 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:35:53.405753    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f9668e_968d_47f5_898d_71fbd3cc64c6.slice/crio-ea83e282af24e1f7f54d0606acc344bee7d0cbba765f6a5dd3b8097d68ad992b WatchSource:0}: Error finding container ea83e282af24e1f7f54d0606acc344bee7d0cbba765f6a5dd3b8097d68ad992b: Status 404 returned error can't find the container with id ea83e282af24e1f7f54d0606acc344bee7d0cbba765f6a5dd3b8097d68ad992b
May 27 23:35:53.438487 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.438456    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf9596469-b2s4d" event={"ID":"eaf47643-d030-4965-9847-13704743ef97","Type":"ContainerStarted","Data":"bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72"}
May 27 23:35:53.438614 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.438492    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf9596469-b2s4d" event={"ID":"eaf47643-d030-4965-9847-13704743ef97","Type":"ContainerStarted","Data":"24799d3c98d5f19b5dbb6f9cd2322decc5ff5851d97b70d4a9d45d07d26ae27f"}
May 27 23:35:53.439452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.439428    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6dcc968c-g4gsk" event={"ID":"e6f9668e-968d-47f5-898d-71fbd3cc64c6","Type":"ContainerStarted","Data":"ea83e282af24e1f7f54d0606acc344bee7d0cbba765f6a5dd3b8097d68ad992b"}
May 27 23:35:53.462325 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:53.462263    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-cf9596469-b2s4d" podStartSLOduration=1.462244532 podStartE2EDuration="1.462244532s" podCreationTimestamp="2026-05-27 23:35:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:35:53.462018482 +0000 UTC m=+144.119880957" watchObservedRunningTime="2026-05-27 23:35:53.462244532 +0000 UTC m=+144.120107010"
May 27 23:35:54.444063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:54.444026    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6dcc968c-g4gsk" event={"ID":"e6f9668e-968d-47f5-898d-71fbd3cc64c6","Type":"ContainerStarted","Data":"ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe"}
May 27 23:35:54.464740 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:54.464681    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6c6dcc968c-g4gsk" podStartSLOduration=2.464659202 podStartE2EDuration="2.464659202s" podCreationTimestamp="2026-05-27 23:35:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:35:54.463031384 +0000 UTC m=+145.120893883" watchObservedRunningTime="2026-05-27 23:35:54.464659202 +0000 UTC m=+145.122521680"
May 27 23:35:58.446548 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:35:58.446498    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-776d4c97f4-hqmvm"
May 27 23:36:02.672248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:02.672210    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:36:02.672248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:02.672259    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:36:02.673428 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:02.673408    2562 patch_prober.go:28] interesting pod/console-cf9596469-b2s4d container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused" start-of-body=
May 27 23:36:02.673478 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:02.673448    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-cf9596469-b2s4d" podUID="eaf47643-d030-4965-9847-13704743ef97" containerName="console" probeResult="failure" output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused"
May 27 23:36:03.271543 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:03.271496    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:36:03.271728 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:03.271573    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:36:03.272969 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:03.272939    2562 patch_prober.go:28] interesting pod/console-6c6dcc968c-g4gsk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.24:8443/health\": dial tcp 10.134.0.24:8443: connect: connection refused" start-of-body=
May 27 23:36:03.273074 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:03.272979    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6c6dcc968c-g4gsk" podUID="e6f9668e-968d-47f5-898d-71fbd3cc64c6" containerName="console" probeResult="failure" output="Get \"https://10.134.0.24:8443/health\": dial tcp 10.134.0.24:8443: connect: connection refused"
May 27 23:36:07.200059 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:36:07.200001    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6knhq" podUID="9c3a8dde-f0f1-4ebe-88ce-f56cd784f813"
May 27 23:36:07.218207 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:36:07.218169    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-hwcgw" podUID="8accd220-388c-4d68-b5d9-f393b8474848"
May 27 23:36:07.483958 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:07.483878    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6knhq"
May 27 23:36:07.484099 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:07.483878    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:36:08.547161 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:08.547129    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:36:08.547558 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:08.547173    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:36:12.132860 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.132772    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:36:12.132860 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.132836    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:36:12.135181 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.135154    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c3a8dde-f0f1-4ebe-88ce-f56cd784f813-metrics-tls\") pod \"dns-default-6knhq\" (UID: \"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813\") " pod="openshift-dns/dns-default-6knhq"
May 27 23:36:12.135304 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.135249    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8accd220-388c-4d68-b5d9-f393b8474848-cert\") pod \"ingress-canary-hwcgw\" (UID: \"8accd220-388c-4d68-b5d9-f393b8474848\") " pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:36:12.287210 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.287181    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lstvz\""
May 27 23:36:12.288085 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.288058    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-szgzp\""
May 27 23:36:12.295124 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.295102    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6knhq"
May 27 23:36:12.295220 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.295174    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwcgw"
May 27 23:36:12.436318 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.436288    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6knhq"]
May 27 23:36:12.438342 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:36:12.438314    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3a8dde_f0f1_4ebe_88ce_f56cd784f813.slice/crio-eb66fa63c2845d2166c7bf31cd06c19276bfb71ff499958f0c2d30f5c54f9d66 WatchSource:0}: Error finding container eb66fa63c2845d2166c7bf31cd06c19276bfb71ff499958f0c2d30f5c54f9d66: Status 404 returned error can't find the container with id eb66fa63c2845d2166c7bf31cd06c19276bfb71ff499958f0c2d30f5c54f9d66
May 27 23:36:12.462691 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.462667    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwcgw"]
May 27 23:36:12.465327 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:36:12.465297    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8accd220_388c_4d68_b5d9_f393b8474848.slice/crio-f7bb4cfbc4137895346f4f6b8e79c3634c6a17b2572556884de4d62f9a12dd2a WatchSource:0}: Error finding container f7bb4cfbc4137895346f4f6b8e79c3634c6a17b2572556884de4d62f9a12dd2a: Status 404 returned error can't find the container with id f7bb4cfbc4137895346f4f6b8e79c3634c6a17b2572556884de4d62f9a12dd2a
May 27 23:36:12.500539 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.500496    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwcgw" event={"ID":"8accd220-388c-4d68-b5d9-f393b8474848","Type":"ContainerStarted","Data":"f7bb4cfbc4137895346f4f6b8e79c3634c6a17b2572556884de4d62f9a12dd2a"}
May 27 23:36:12.501416 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.501389    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6knhq" event={"ID":"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813","Type":"ContainerStarted","Data":"eb66fa63c2845d2166c7bf31cd06c19276bfb71ff499958f0c2d30f5c54f9d66"}
May 27 23:36:12.676686 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.676606    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:36:12.680596 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:12.680572    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:36:13.277290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:13.277183    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:36:13.282862 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:13.282836    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:36:13.344860 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:13.344829    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cf9596469-b2s4d"]
May 27 23:36:15.512859 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:15.512820    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwcgw" event={"ID":"8accd220-388c-4d68-b5d9-f393b8474848","Type":"ContainerStarted","Data":"61477d34c29e883d999f85fab6c3f5472fcfca41b9f4a452c747b62dba8f4e28"}
May 27 23:36:15.514402 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:15.514375    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6knhq" event={"ID":"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813","Type":"ContainerStarted","Data":"d72dbe30fb189ee845415d0c9751b7fcde62144e8b62fd0053cd1a945747a64b"}
May 27 23:36:15.514402 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:15.514401    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6knhq" event={"ID":"9c3a8dde-f0f1-4ebe-88ce-f56cd784f813","Type":"ContainerStarted","Data":"5eb11527d2947be11a051b0195659182ef42e1cc4ca9169241aa2d65ba6a5c02"}
May 27 23:36:15.514618 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:15.514494    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6knhq"
May 27 23:36:15.529551 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:15.529472    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hwcgw" podStartSLOduration=129.460744118 podStartE2EDuration="2m11.529453644s" podCreationTimestamp="2026-05-27 23:34:04 +0000 UTC" firstStartedPulling="2026-05-27 23:36:12.467232128 +0000 UTC m=+163.125094580" lastFinishedPulling="2026-05-27 23:36:14.535941654 +0000 UTC m=+165.193804106" observedRunningTime="2026-05-27 23:36:15.528365728 +0000 UTC m=+166.186228214" watchObservedRunningTime="2026-05-27 23:36:15.529453644 +0000 UTC m=+166.187316118"
May 27 23:36:15.547128 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:15.547067    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-6knhq" podStartSLOduration=129.455512756 podStartE2EDuration="2m11.547046867s" podCreationTimestamp="2026-05-27 23:34:04 +0000 UTC" firstStartedPulling="2026-05-27 23:36:12.440983897 +0000 UTC m=+163.098846349" lastFinishedPulling="2026-05-27 23:36:14.532518005 +0000 UTC m=+165.190380460" observedRunningTime="2026-05-27 23:36:15.544567373 +0000 UTC m=+166.202429844" watchObservedRunningTime="2026-05-27 23:36:15.547046867 +0000 UTC m=+166.204909342"
May 27 23:36:17.338569 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.338504    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6fdf79cb5c-hwdg9" podUID="ac05b693-5665-41ba-8647-9b49ad5e7ebe" containerName="console" containerID="cri-o://6c7227543dad8def517c0fb5e8c7638fff3cb4fff953409ac88f1b2b90c4bdb2" gracePeriod=15
May 27 23:36:17.523869 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.523836    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fdf79cb5c-hwdg9_ac05b693-5665-41ba-8647-9b49ad5e7ebe/console/0.log"
May 27 23:36:17.524050 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.523876    2562 generic.go:358] "Generic (PLEG): container finished" podID="ac05b693-5665-41ba-8647-9b49ad5e7ebe" containerID="6c7227543dad8def517c0fb5e8c7638fff3cb4fff953409ac88f1b2b90c4bdb2" exitCode=2
May 27 23:36:17.524050 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.523966    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdf79cb5c-hwdg9" event={"ID":"ac05b693-5665-41ba-8647-9b49ad5e7ebe","Type":"ContainerDied","Data":"6c7227543dad8def517c0fb5e8c7638fff3cb4fff953409ac88f1b2b90c4bdb2"}
May 27 23:36:17.587961 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.587941    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fdf79cb5c-hwdg9_ac05b693-5665-41ba-8647-9b49ad5e7ebe/console/0.log"
May 27 23:36:17.588117 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.588011    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:36:17.687197 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.687102    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-serving-cert\") pod \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") "
May 27 23:36:17.687197 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.687153    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-oauth-serving-cert\") pod \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") "
May 27 23:36:17.687197 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.687175    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-config\") pod \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") "
May 27 23:36:17.687477 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.687206    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxzjm\" (UniqueName: \"kubernetes.io/projected/ac05b693-5665-41ba-8647-9b49ad5e7ebe-kube-api-access-bxzjm\") pod \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") "
May 27 23:36:17.687477 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.687261    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-service-ca\") pod \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") "
May 27 23:36:17.687477 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.687286    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-oauth-config\") pod \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\" (UID: \"ac05b693-5665-41ba-8647-9b49ad5e7ebe\") "
May 27 23:36:17.687705 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.687677    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ac05b693-5665-41ba-8647-9b49ad5e7ebe" (UID: "ac05b693-5665-41ba-8647-9b49ad5e7ebe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:36:17.687705 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.687691    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-config" (OuterVolumeSpecName: "console-config") pod "ac05b693-5665-41ba-8647-9b49ad5e7ebe" (UID: "ac05b693-5665-41ba-8647-9b49ad5e7ebe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:36:17.687854 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.687696    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-service-ca" (OuterVolumeSpecName: "service-ca") pod "ac05b693-5665-41ba-8647-9b49ad5e7ebe" (UID: "ac05b693-5665-41ba-8647-9b49ad5e7ebe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:36:17.689474 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.689441    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ac05b693-5665-41ba-8647-9b49ad5e7ebe" (UID: "ac05b693-5665-41ba-8647-9b49ad5e7ebe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:36:17.689474 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.689463    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ac05b693-5665-41ba-8647-9b49ad5e7ebe" (UID: "ac05b693-5665-41ba-8647-9b49ad5e7ebe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:36:17.689622 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.689499    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac05b693-5665-41ba-8647-9b49ad5e7ebe-kube-api-access-bxzjm" (OuterVolumeSpecName: "kube-api-access-bxzjm") pod "ac05b693-5665-41ba-8647-9b49ad5e7ebe" (UID: "ac05b693-5665-41ba-8647-9b49ad5e7ebe"). InnerVolumeSpecName "kube-api-access-bxzjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:36:17.788255 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.788202    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-serving-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:17.788255 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.788249    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-oauth-serving-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:17.788255 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.788260    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-config\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:17.788255 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.788269    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxzjm\" (UniqueName: \"kubernetes.io/projected/ac05b693-5665-41ba-8647-9b49ad5e7ebe-kube-api-access-bxzjm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:17.788520 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.788281    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac05b693-5665-41ba-8647-9b49ad5e7ebe-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:17.788520 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:17.788289    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac05b693-5665-41ba-8647-9b49ad5e7ebe-console-oauth-config\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:18.528002 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:18.527976    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fdf79cb5c-hwdg9_ac05b693-5665-41ba-8647-9b49ad5e7ebe/console/0.log"
May 27 23:36:18.528417 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:18.528078    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdf79cb5c-hwdg9" event={"ID":"ac05b693-5665-41ba-8647-9b49ad5e7ebe","Type":"ContainerDied","Data":"40d72a8fba8523b57db89a707e928272ed824bf2950d4a14fb455155fb862557"}
May 27 23:36:18.528417 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:18.528104    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdf79cb5c-hwdg9"
May 27 23:36:18.528417 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:18.528119    2562 scope.go:117] "RemoveContainer" containerID="6c7227543dad8def517c0fb5e8c7638fff3cb4fff953409ac88f1b2b90c4bdb2"
May 27 23:36:18.548492 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:18.548463    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fdf79cb5c-hwdg9"]
May 27 23:36:18.552127 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:18.552098    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fdf79cb5c-hwdg9"]
May 27 23:36:19.944655 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:19.944619    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac05b693-5665-41ba-8647-9b49ad5e7ebe" path="/var/lib/kubelet/pods/ac05b693-5665-41ba-8647-9b49ad5e7ebe/volumes"
May 27 23:36:22.542969 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:22.542932    2562 generic.go:358] "Generic (PLEG): container finished" podID="56d6868c-dba0-46ce-aaa0-78324081e3e4" containerID="aed330a283a4bf6f4f6e26f75af8dd8f30e412fe02fab94d1191299c7780f637" exitCode=0
May 27 23:36:22.543359 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:22.543006    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-gtr8n" event={"ID":"56d6868c-dba0-46ce-aaa0-78324081e3e4","Type":"ContainerDied","Data":"aed330a283a4bf6f4f6e26f75af8dd8f30e412fe02fab94d1191299c7780f637"}
May 27 23:36:22.543399 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:22.543376    2562 scope.go:117] "RemoveContainer" containerID="aed330a283a4bf6f4f6e26f75af8dd8f30e412fe02fab94d1191299c7780f637"
May 27 23:36:23.223691 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:23.223660    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-749c9c75f6-f7z9x_303186fe-da16-4eef-9f9b-6406b1ae5d04/router/0.log"
May 27 23:36:23.395333 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:23.395306    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hwcgw_8accd220-388c-4d68-b5d9-f393b8474848/serve-healthcheck-canary/0.log"
May 27 23:36:23.547325 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:23.547288    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-gtr8n" event={"ID":"56d6868c-dba0-46ce-aaa0-78324081e3e4","Type":"ContainerStarted","Data":"f6c1960280c6071dbcd178a5b3af8d404d1c4dfb67b87187ae187ff90de6da19"}
May 27 23:36:23.796051 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:23.796018    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-2rgz7_61e303c0-6b4f-437a-8f22-17e89a23c62b/cluster-samples-operator/0.log"
May 27 23:36:23.994966 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:23.994943    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-2rgz7_61e303c0-6b4f-437a-8f22-17e89a23c62b/cluster-samples-operator-watch/0.log"
May 27 23:36:25.520910 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:25.520876    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6knhq"
May 27 23:36:28.552172 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:28.552141    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:36:28.556154 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:28.556132    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f94fbfbcf-hpgxp"
May 27 23:36:39.530022 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.529978    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-cf9596469-b2s4d" podUID="eaf47643-d030-4965-9847-13704743ef97" containerName="console" containerID="cri-o://bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72" gracePeriod=15
May 27 23:36:39.786754 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.786689    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-cf9596469-b2s4d_eaf47643-d030-4965-9847-13704743ef97/console/0.log"
May 27 23:36:39.786754 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.786753    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:36:39.882513 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.882468    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-oauth-serving-cert\") pod \"eaf47643-d030-4965-9847-13704743ef97\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") "
May 27 23:36:39.882711 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.882555    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-oauth-config\") pod \"eaf47643-d030-4965-9847-13704743ef97\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") "
May 27 23:36:39.882711 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.882590    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-console-config\") pod \"eaf47643-d030-4965-9847-13704743ef97\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") "
May 27 23:36:39.882711 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.882650    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvlgf\" (UniqueName: \"kubernetes.io/projected/eaf47643-d030-4965-9847-13704743ef97-kube-api-access-pvlgf\") pod \"eaf47643-d030-4965-9847-13704743ef97\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") "
May 27 23:36:39.882711 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.882683    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-serving-cert\") pod \"eaf47643-d030-4965-9847-13704743ef97\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") "
May 27 23:36:39.882927 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.882729    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-service-ca\") pod \"eaf47643-d030-4965-9847-13704743ef97\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") "
May 27 23:36:39.882927 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.882774    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-trusted-ca-bundle\") pod \"eaf47643-d030-4965-9847-13704743ef97\" (UID: \"eaf47643-d030-4965-9847-13704743ef97\") "
May 27 23:36:39.883066 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.883035    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eaf47643-d030-4965-9847-13704743ef97" (UID: "eaf47643-d030-4965-9847-13704743ef97"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:36:39.883221 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.883185    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-service-ca" (OuterVolumeSpecName: "service-ca") pod "eaf47643-d030-4965-9847-13704743ef97" (UID: "eaf47643-d030-4965-9847-13704743ef97"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:36:39.883349 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.883270    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eaf47643-d030-4965-9847-13704743ef97" (UID: "eaf47643-d030-4965-9847-13704743ef97"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:36:39.883445 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.883424    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-console-config" (OuterVolumeSpecName: "console-config") pod "eaf47643-d030-4965-9847-13704743ef97" (UID: "eaf47643-d030-4965-9847-13704743ef97"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:36:39.884860 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.884831    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eaf47643-d030-4965-9847-13704743ef97" (UID: "eaf47643-d030-4965-9847-13704743ef97"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:36:39.884940 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.884868    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf47643-d030-4965-9847-13704743ef97-kube-api-access-pvlgf" (OuterVolumeSpecName: "kube-api-access-pvlgf") pod "eaf47643-d030-4965-9847-13704743ef97" (UID: "eaf47643-d030-4965-9847-13704743ef97"). InnerVolumeSpecName "kube-api-access-pvlgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:36:39.884940 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.884900    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eaf47643-d030-4965-9847-13704743ef97" (UID: "eaf47643-d030-4965-9847-13704743ef97"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:36:39.984478 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.984447    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-trusted-ca-bundle\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:39.984478 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.984478    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-oauth-serving-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:39.984715 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.984493    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-oauth-config\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:39.984715 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.984505    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-console-config\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:39.984715 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.984518    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvlgf\" (UniqueName: \"kubernetes.io/projected/eaf47643-d030-4965-9847-13704743ef97-kube-api-access-pvlgf\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:39.984715 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.984545    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eaf47643-d030-4965-9847-13704743ef97-console-serving-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:39.984715 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:39.984557    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf47643-d030-4965-9847-13704743ef97-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:36:40.605898 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.605869    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-cf9596469-b2s4d_eaf47643-d030-4965-9847-13704743ef97/console/0.log"
May 27 23:36:40.606296 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.605915    2562 generic.go:358] "Generic (PLEG): container finished" podID="eaf47643-d030-4965-9847-13704743ef97" containerID="bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72" exitCode=2
May 27 23:36:40.606296 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.606008    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf9596469-b2s4d" event={"ID":"eaf47643-d030-4965-9847-13704743ef97","Type":"ContainerDied","Data":"bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72"}
May 27 23:36:40.606296 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.606047    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf9596469-b2s4d" event={"ID":"eaf47643-d030-4965-9847-13704743ef97","Type":"ContainerDied","Data":"24799d3c98d5f19b5dbb6f9cd2322decc5ff5851d97b70d4a9d45d07d26ae27f"}
May 27 23:36:40.606296 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.606063    2562 scope.go:117] "RemoveContainer" containerID="bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72"
May 27 23:36:40.606296 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.606018    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf9596469-b2s4d"
May 27 23:36:40.615171 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.615154    2562 scope.go:117] "RemoveContainer" containerID="bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72"
May 27 23:36:40.615468 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:36:40.615445    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72\": container with ID starting with bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72 not found: ID does not exist" containerID="bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72"
May 27 23:36:40.615519 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.615478    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72"} err="failed to get container status \"bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72\": rpc error: code = NotFound desc = could not find container \"bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72\": container with ID starting with bcfd190daaa480593c0372605c1e2803aa35785a61ab60b2ce8687facc35df72 not found: ID does not exist"
May 27 23:36:40.625456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.625428    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cf9596469-b2s4d"]
May 27 23:36:40.630371 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:40.630342    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cf9596469-b2s4d"]
May 27 23:36:41.944738 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:36:41.944700    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaf47643-d030-4965-9847-13704743ef97" path="/var/lib/kubelet/pods/eaf47643-d030-4965-9847-13704743ef97/volumes"
May 27 23:37:21.756055 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.756023    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vrcnk"]
May 27 23:37:21.756507 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.756387    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac05b693-5665-41ba-8647-9b49ad5e7ebe" containerName="console"
May 27 23:37:21.756507 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.756399    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac05b693-5665-41ba-8647-9b49ad5e7ebe" containerName="console"
May 27 23:37:21.756507 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.756416    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf47643-d030-4965-9847-13704743ef97" containerName="console"
May 27 23:37:21.756507 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.756421    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf47643-d030-4965-9847-13704743ef97" containerName="console"
May 27 23:37:21.756507 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.756474    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="eaf47643-d030-4965-9847-13704743ef97" containerName="console"
May 27 23:37:21.756507 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.756487    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="ac05b693-5665-41ba-8647-9b49ad5e7ebe" containerName="console"
May 27 23:37:21.759393 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.759377    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:21.763961 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.763940    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 27 23:37:21.775611 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.775583    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vrcnk"]
May 27 23:37:21.866171 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.866131    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-58dd6bbb87-dxzq5"]
May 27 23:37:21.869621 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.869599    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:21.894364 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.894328    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58dd6bbb87-dxzq5"]
May 27 23:37:21.949728 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.949681    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9e31fb0-0d7c-4096-8e90-990753c8c523-original-pull-secret\") pod \"global-pull-secret-syncer-vrcnk\" (UID: \"e9e31fb0-0d7c-4096-8e90-990753c8c523\") " pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:21.949914 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.949751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9e31fb0-0d7c-4096-8e90-990753c8c523-dbus\") pod \"global-pull-secret-syncer-vrcnk\" (UID: \"e9e31fb0-0d7c-4096-8e90-990753c8c523\") " pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:21.949914 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:21.949773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9e31fb0-0d7c-4096-8e90-990753c8c523-kubelet-config\") pod \"global-pull-secret-syncer-vrcnk\" (UID: \"e9e31fb0-0d7c-4096-8e90-990753c8c523\") " pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:22.050783 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-serving-cert\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.050783 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050730    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-trusted-ca-bundle\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.050783 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-config\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.051063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050804    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9e31fb0-0d7c-4096-8e90-990753c8c523-original-pull-secret\") pod \"global-pull-secret-syncer-vrcnk\" (UID: \"e9e31fb0-0d7c-4096-8e90-990753c8c523\") " pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:22.051063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-service-ca\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.051063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050868    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9e31fb0-0d7c-4096-8e90-990753c8c523-dbus\") pod \"global-pull-secret-syncer-vrcnk\" (UID: \"e9e31fb0-0d7c-4096-8e90-990753c8c523\") " pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:22.051063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-oauth-serving-cert\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.051063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050906    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-oauth-config\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.051063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050936    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9e31fb0-0d7c-4096-8e90-990753c8c523-kubelet-config\") pod \"global-pull-secret-syncer-vrcnk\" (UID: \"e9e31fb0-0d7c-4096-8e90-990753c8c523\") " pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:22.051063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.050960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hqd\" (UniqueName: \"kubernetes.io/projected/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-kube-api-access-q6hqd\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.051063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.051017    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9e31fb0-0d7c-4096-8e90-990753c8c523-kubelet-config\") pod \"global-pull-secret-syncer-vrcnk\" (UID: \"e9e31fb0-0d7c-4096-8e90-990753c8c523\") " pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:22.051063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.051038    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9e31fb0-0d7c-4096-8e90-990753c8c523-dbus\") pod \"global-pull-secret-syncer-vrcnk\" (UID: \"e9e31fb0-0d7c-4096-8e90-990753c8c523\") " pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:22.053121 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.053094    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9e31fb0-0d7c-4096-8e90-990753c8c523-original-pull-secret\") pod \"global-pull-secret-syncer-vrcnk\" (UID: \"e9e31fb0-0d7c-4096-8e90-990753c8c523\") " pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:22.069100 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.069072    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vrcnk"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.151796    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-service-ca\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.151900    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-oauth-serving-cert\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.151977    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-oauth-config\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.152011    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hqd\" (UniqueName: \"kubernetes.io/projected/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-kube-api-access-q6hqd\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.152061    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-serving-cert\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.152109    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-trusted-ca-bundle\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.152152    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-config\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.152599    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-oauth-serving-cert\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.152951    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-config\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.153319 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.153020    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-service-ca\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.154010 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.153756    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-trusted-ca-bundle\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.156655 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.156633    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-serving-cert\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.157387 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.157363    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-oauth-config\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.166556 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.166505    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hqd\" (UniqueName: \"kubernetes.io/projected/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-kube-api-access-q6hqd\") pod \"console-58dd6bbb87-dxzq5\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") " pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.179215 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.179176    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:22.213817 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.213693    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vrcnk"]
May 27 23:37:22.311283 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.311242    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58dd6bbb87-dxzq5"]
May 27 23:37:22.313968 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:37:22.313936    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60cc25d_c6f9_442b_a4d6_debcd9e353bf.slice/crio-4c0d3ac53a619a282b13e315719f086022422381b6fd9f0a5c72c0b4d49a43d9 WatchSource:0}: Error finding container 4c0d3ac53a619a282b13e315719f086022422381b6fd9f0a5c72c0b4d49a43d9: Status 404 returned error can't find the container with id 4c0d3ac53a619a282b13e315719f086022422381b6fd9f0a5c72c0b4d49a43d9
May 27 23:37:22.736910 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.736818    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58dd6bbb87-dxzq5" event={"ID":"e60cc25d-c6f9-442b-a4d6-debcd9e353bf","Type":"ContainerStarted","Data":"370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9"}
May 27 23:37:22.736910 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.736852    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58dd6bbb87-dxzq5" event={"ID":"e60cc25d-c6f9-442b-a4d6-debcd9e353bf","Type":"ContainerStarted","Data":"4c0d3ac53a619a282b13e315719f086022422381b6fd9f0a5c72c0b4d49a43d9"}
May 27 23:37:22.737906 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.737878    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vrcnk" event={"ID":"e9e31fb0-0d7c-4096-8e90-990753c8c523","Type":"ContainerStarted","Data":"3e8120bd6e80fe818d35a7e992eedb0317617e48397640a4eff9b4864e2af387"}
May 27 23:37:22.758750 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:22.758697    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-58dd6bbb87-dxzq5" podStartSLOduration=1.7586830359999999 podStartE2EDuration="1.758683036s" podCreationTimestamp="2026-05-27 23:37:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:37:22.755916535 +0000 UTC m=+233.413779048" watchObservedRunningTime="2026-05-27 23:37:22.758683036 +0000 UTC m=+233.416545504"
May 27 23:37:26.753602 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:26.753563    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vrcnk" event={"ID":"e9e31fb0-0d7c-4096-8e90-990753c8c523","Type":"ContainerStarted","Data":"12edd393bcb81b286f63e52722eb3b424d21d1a3835b4a9c63947bc3b25b1023"}
May 27 23:37:26.772827 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:26.772765    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vrcnk" podStartSLOduration=1.355656297 podStartE2EDuration="5.772749091s" podCreationTimestamp="2026-05-27 23:37:21 +0000 UTC" firstStartedPulling="2026-05-27 23:37:22.218576129 +0000 UTC m=+232.876438585" lastFinishedPulling="2026-05-27 23:37:26.635668924 +0000 UTC m=+237.293531379" observedRunningTime="2026-05-27 23:37:26.770974138 +0000 UTC m=+237.428836641" watchObservedRunningTime="2026-05-27 23:37:26.772749091 +0000 UTC m=+237.430611606"
May 27 23:37:32.180039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:32.179999    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:32.180039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:32.180050    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:32.184891 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:32.184868    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:32.774122 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:32.774098    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:37:32.838952 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:32.838908    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6dcc968c-g4gsk"]
May 27 23:37:57.865101 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:57.865001    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6c6dcc968c-g4gsk" podUID="e6f9668e-968d-47f5-898d-71fbd3cc64c6" containerName="console" containerID="cri-o://ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe" gracePeriod=15
May 27 23:37:58.123737 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.123676    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c6dcc968c-g4gsk_e6f9668e-968d-47f5-898d-71fbd3cc64c6/console/0.log"
May 27 23:37:58.123855 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.123739    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:37:58.265703 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.265666    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-config\") pod \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") "
May 27 23:37:58.265703 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.265710    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-service-ca\") pod \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") "
May 27 23:37:58.265947 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.265743    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-trusted-ca-bundle\") pod \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") "
May 27 23:37:58.265947 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.265788    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-serving-cert\") pod \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") "
May 27 23:37:58.265947 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.265815    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-oauth-serving-cert\") pod \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") "
May 27 23:37:58.265947 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.265841    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4c4g\" (UniqueName: \"kubernetes.io/projected/e6f9668e-968d-47f5-898d-71fbd3cc64c6-kube-api-access-s4c4g\") pod \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") "
May 27 23:37:58.265947 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.265880    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-oauth-config\") pod \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\" (UID: \"e6f9668e-968d-47f5-898d-71fbd3cc64c6\") "
May 27 23:37:58.266183 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.266148    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-config" (OuterVolumeSpecName: "console-config") pod "e6f9668e-968d-47f5-898d-71fbd3cc64c6" (UID: "e6f9668e-968d-47f5-898d-71fbd3cc64c6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:37:58.266293 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.266215    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-service-ca" (OuterVolumeSpecName: "service-ca") pod "e6f9668e-968d-47f5-898d-71fbd3cc64c6" (UID: "e6f9668e-968d-47f5-898d-71fbd3cc64c6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:37:58.266293 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.266238    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e6f9668e-968d-47f5-898d-71fbd3cc64c6" (UID: "e6f9668e-968d-47f5-898d-71fbd3cc64c6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:37:58.266465 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.266443    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e6f9668e-968d-47f5-898d-71fbd3cc64c6" (UID: "e6f9668e-968d-47f5-898d-71fbd3cc64c6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:37:58.268220 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.268189    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e6f9668e-968d-47f5-898d-71fbd3cc64c6" (UID: "e6f9668e-968d-47f5-898d-71fbd3cc64c6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:37:58.268220 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.268190    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f9668e-968d-47f5-898d-71fbd3cc64c6-kube-api-access-s4c4g" (OuterVolumeSpecName: "kube-api-access-s4c4g") pod "e6f9668e-968d-47f5-898d-71fbd3cc64c6" (UID: "e6f9668e-968d-47f5-898d-71fbd3cc64c6"). InnerVolumeSpecName "kube-api-access-s4c4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:37:58.268364 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.268258    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e6f9668e-968d-47f5-898d-71fbd3cc64c6" (UID: "e6f9668e-968d-47f5-898d-71fbd3cc64c6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:37:58.367106 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.367066    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-config\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:37:58.367106 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.367098    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:37:58.367106 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.367108    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-trusted-ca-bundle\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:37:58.367347 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.367118    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-serving-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:37:58.367347 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.367127    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6f9668e-968d-47f5-898d-71fbd3cc64c6-oauth-serving-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:37:58.367347 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.367138    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4c4g\" (UniqueName: \"kubernetes.io/projected/e6f9668e-968d-47f5-898d-71fbd3cc64c6-kube-api-access-s4c4g\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:37:58.367347 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.367146    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6f9668e-968d-47f5-898d-71fbd3cc64c6-console-oauth-config\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:37:58.849978 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.849935    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c6dcc968c-g4gsk_e6f9668e-968d-47f5-898d-71fbd3cc64c6/console/0.log"
May 27 23:37:58.850155 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.850000    2562 generic.go:358] "Generic (PLEG): container finished" podID="e6f9668e-968d-47f5-898d-71fbd3cc64c6" containerID="ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe" exitCode=2
May 27 23:37:58.850155 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.850076    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6dcc968c-g4gsk"
May 27 23:37:58.850155 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.850096    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6dcc968c-g4gsk" event={"ID":"e6f9668e-968d-47f5-898d-71fbd3cc64c6","Type":"ContainerDied","Data":"ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe"}
May 27 23:37:58.850155 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.850139    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6dcc968c-g4gsk" event={"ID":"e6f9668e-968d-47f5-898d-71fbd3cc64c6","Type":"ContainerDied","Data":"ea83e282af24e1f7f54d0606acc344bee7d0cbba765f6a5dd3b8097d68ad992b"}
May 27 23:37:58.850293 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.850160    2562 scope.go:117] "RemoveContainer" containerID="ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe"
May 27 23:37:58.858711 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.858691    2562 scope.go:117] "RemoveContainer" containerID="ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe"
May 27 23:37:58.858982 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:37:58.858961    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe\": container with ID starting with ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe not found: ID does not exist" containerID="ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe"
May 27 23:37:58.859058 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.858990    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe"} err="failed to get container status \"ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe\": rpc error: code = NotFound desc = could not find container \"ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe\": container with ID starting with ac20ec4b2786208575166756be6f1fd230aafaea07b23d1287b90e5b5030d1fe not found: ID does not exist"
May 27 23:37:58.886067 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.886026    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6dcc968c-g4gsk"]
May 27 23:37:58.892730 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:58.892703    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c6dcc968c-g4gsk"]
May 27 23:37:59.944417 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:37:59.944384    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6f9668e-968d-47f5-898d-71fbd3cc64c6" path="/var/lib/kubelet/pods/e6f9668e-968d-47f5-898d-71fbd3cc64c6/volumes"
May 27 23:38:29.857946 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:29.857909    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:38:29.860171 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:29.860147    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:38:29.864415 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:29.864396    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:38:29.866434 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:29.866412    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:38:29.868811 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:29.868794    2562 kubelet.go:1628] "Image garbage collection succeeded"
May 27 23:38:57.742283 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.742243    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"]
May 27 23:38:57.744665 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.742602    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f9668e-968d-47f5-898d-71fbd3cc64c6" containerName="console"
May 27 23:38:57.744665 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.742613    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f9668e-968d-47f5-898d-71fbd3cc64c6" containerName="console"
May 27 23:38:57.744665 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.742686    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6f9668e-968d-47f5-898d-71fbd3cc64c6" containerName="console"
May 27 23:38:57.745602 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.745585    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"
May 27 23:38:57.750063 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.750022    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 27 23:38:57.750214 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.750144    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-jtc89\""
May 27 23:38:57.752561 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.752222    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 27 23:38:57.756434 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.756410    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"]
May 27 23:38:57.891854 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.891818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfz8n\" (UniqueName: \"kubernetes.io/projected/ededa324-b6a1-4f96-ae99-f517ffd1d815-kube-api-access-wfz8n\") pod \"feast-operator-controller-manager-5d47bdf4cc-fs226\" (UID: \"ededa324-b6a1-4f96-ae99-f517ffd1d815\") " pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"
May 27 23:38:57.992674 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:57.992586    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wfz8n\" (UniqueName: \"kubernetes.io/projected/ededa324-b6a1-4f96-ae99-f517ffd1d815-kube-api-access-wfz8n\") pod \"feast-operator-controller-manager-5d47bdf4cc-fs226\" (UID: \"ededa324-b6a1-4f96-ae99-f517ffd1d815\") " pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"
May 27 23:38:58.003762 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:58.003730    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfz8n\" (UniqueName: \"kubernetes.io/projected/ededa324-b6a1-4f96-ae99-f517ffd1d815-kube-api-access-wfz8n\") pod \"feast-operator-controller-manager-5d47bdf4cc-fs226\" (UID: \"ededa324-b6a1-4f96-ae99-f517ffd1d815\") " pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"
May 27 23:38:58.056544 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:58.056502    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"
May 27 23:38:58.191408 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:58.191354    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"]
May 27 23:38:58.193745 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:38:58.193714    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podededa324_b6a1_4f96_ae99_f517ffd1d815.slice/crio-616ce686ca9f9a56dd18860bb7cdc86664896d7cdc59f31ee69fe474de44d036 WatchSource:0}: Error finding container 616ce686ca9f9a56dd18860bb7cdc86664896d7cdc59f31ee69fe474de44d036: Status 404 returned error can't find the container with id 616ce686ca9f9a56dd18860bb7cdc86664896d7cdc59f31ee69fe474de44d036
May 27 23:38:58.195392 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:58.195375    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 23:38:59.030059 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:38:59.030013    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226" event={"ID":"ededa324-b6a1-4f96-ae99-f517ffd1d815","Type":"ContainerStarted","Data":"616ce686ca9f9a56dd18860bb7cdc86664896d7cdc59f31ee69fe474de44d036"}
May 27 23:39:01.038324 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:01.038284    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226" event={"ID":"ededa324-b6a1-4f96-ae99-f517ffd1d815","Type":"ContainerStarted","Data":"c2917ae7b6bc0d0c2791b3f1da1940e4c23c33b1ac78f841bb0e1b0a5d43a07e"}
May 27 23:39:01.038934 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:01.038396    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"
May 27 23:39:01.065625 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:01.065573    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226" podStartSLOduration=1.550425382 podStartE2EDuration="4.065554905s" podCreationTimestamp="2026-05-27 23:38:57 +0000 UTC" firstStartedPulling="2026-05-27 23:38:58.195556063 +0000 UTC m=+328.853418515" lastFinishedPulling="2026-05-27 23:39:00.710685568 +0000 UTC m=+331.368548038" observedRunningTime="2026-05-27 23:39:01.064599635 +0000 UTC m=+331.722462109" watchObservedRunningTime="2026-05-27 23:39:01.065554905 +0000 UTC m=+331.723417378"
May 27 23:39:12.043863 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:12.043778    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"
May 27 23:39:39.660764 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.660728    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"]
May 27 23:39:39.669469 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.669448    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.672689 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.672667    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 27 23:39:39.673100 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.672987    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 27 23:39:39.673234 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.673216    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 27 23:39:39.673298 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.673231    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 27 23:39:39.673626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.673402    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 27 23:39:39.673626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.673464    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-6kfqc\""
May 27 23:39:39.673851 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.673662    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 27 23:39:39.673851 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.673736    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 27 23:39:39.674694 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.674674    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"]
May 27 23:39:39.759179 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.759144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wdvh\" (UniqueName: \"kubernetes.io/projected/e3f8a371-e095-42cc-a14d-2e4b68af3d29-kube-api-access-9wdvh\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.759179 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.759185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f8a371-e095-42cc-a14d-2e4b68af3d29-feast-data\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.759385 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.759291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-offline-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.759385 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.759317    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.759385 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.759337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-ui-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.759481 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.759448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-registry-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.860306 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.860270    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-offline-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.860306 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.860308    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.860572 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.860323    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-ui-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.860572 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.860372    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-registry-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.860572 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.860421    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9wdvh\" (UniqueName: \"kubernetes.io/projected/e3f8a371-e095-42cc-a14d-2e4b68af3d29-kube-api-access-9wdvh\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.860572 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:39:39.860428    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 27 23:39:39.860572 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.860453    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f8a371-e095-42cc-a14d-2e4b68af3d29-feast-data\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.860572 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:39:39.860500    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 27 23:39:39.860572 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:39:39.860555    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls podName:e3f8a371-e095-42cc-a14d-2e4b68af3d29 nodeName:}" failed. No retries permitted until 2026-05-27 23:39:40.360503488 +0000 UTC m=+371.018365951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls") pod "feast-simple-feast-setup-56d797bf78-m6fph" (UID: "e3f8a371-e095-42cc-a14d-2e4b68af3d29") : secret "feast-simple-feast-setup-online-tls" not found
May 27 23:39:39.860933 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:39:39.860590    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-ui-tls podName:e3f8a371-e095-42cc-a14d-2e4b68af3d29 nodeName:}" failed. No retries permitted until 2026-05-27 23:39:40.360571268 +0000 UTC m=+371.018433722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-ui-tls") pod "feast-simple-feast-setup-56d797bf78-m6fph" (UID: "e3f8a371-e095-42cc-a14d-2e4b68af3d29") : secret "feast-simple-feast-setup-ui-tls" not found
May 27 23:39:39.860933 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.860852    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f8a371-e095-42cc-a14d-2e4b68af3d29-feast-data\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.862812 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.862791    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-registry-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.862960 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.862922    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-offline-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:39.871221 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:39.871197    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wdvh\" (UniqueName: \"kubernetes.io/projected/e3f8a371-e095-42cc-a14d-2e4b68af3d29-kube-api-access-9wdvh\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:40.364912 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:40.364877    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:40.364912 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:40.364912    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-ui-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:40.365123 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:39:40.365023    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 27 23:39:40.365123 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:39:40.365100    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls podName:e3f8a371-e095-42cc-a14d-2e4b68af3d29 nodeName:}" failed. No retries permitted until 2026-05-27 23:39:41.365084285 +0000 UTC m=+372.022946742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls") pod "feast-simple-feast-setup-56d797bf78-m6fph" (UID: "e3f8a371-e095-42cc-a14d-2e4b68af3d29") : secret "feast-simple-feast-setup-online-tls" not found
May 27 23:39:40.367282 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:40.367258    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-ui-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:41.375055 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:41.375014    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:41.377431 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:41.377407    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls\") pod \"feast-simple-feast-setup-56d797bf78-m6fph\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:41.493421 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:41.493391    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:39:41.631734 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:41.631707    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"]
May 27 23:39:41.632753 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:39:41.632730    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3f8a371_e095_42cc_a14d_2e4b68af3d29.slice/crio-9108c55c076d60dc212e9ce99662c27d89bfbd6e1f6346bc598069b3a4977275 WatchSource:0}: Error finding container 9108c55c076d60dc212e9ce99662c27d89bfbd6e1f6346bc598069b3a4977275: Status 404 returned error can't find the container with id 9108c55c076d60dc212e9ce99662c27d89bfbd6e1f6346bc598069b3a4977275
May 27 23:39:42.170520 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:39:42.170474    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerStarted","Data":"9108c55c076d60dc212e9ce99662c27d89bfbd6e1f6346bc598069b3a4977275"}
May 27 23:40:10.265837 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:10.265798    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerStarted","Data":"49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce"}
May 27 23:40:14.280232 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:14.280195    2562 generic.go:358] "Generic (PLEG): container finished" podID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerID="49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce" exitCode=0
May 27 23:40:14.280621 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:14.280263    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerDied","Data":"49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce"}
May 27 23:40:15.285234 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:15.285197    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerStarted","Data":"f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad"}
May 27 23:40:21.307201 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:21.307162    2562 generic.go:358] "Generic (PLEG): container finished" podID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerID="f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad" exitCode=0
May 27 23:40:21.307585 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:21.307210    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerDied","Data":"f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad"}
May 27 23:40:22.315049 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:22.315003    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerStarted","Data":"4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8"}
May 27 23:40:22.315049 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:22.315053    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerStarted","Data":"2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d"}
May 27 23:40:22.315669 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:22.315066    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerStarted","Data":"f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5"}
May 27 23:40:22.315669 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:22.315078    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerStarted","Data":"474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d"}
May 27 23:40:22.344672 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:22.342354    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podStartSLOduration=15.269441006 podStartE2EDuration="43.342332961s" podCreationTimestamp="2026-05-27 23:39:39 +0000 UTC" firstStartedPulling="2026-05-27 23:39:41.634657346 +0000 UTC m=+372.292519798" lastFinishedPulling="2026-05-27 23:40:09.707549298 +0000 UTC m=+400.365411753" observedRunningTime="2026-05-27 23:40:22.340795142 +0000 UTC m=+412.998657616" watchObservedRunningTime="2026-05-27 23:40:22.342332961 +0000 UTC m=+413.000195437"
May 27 23:40:23.494415 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:23.494370    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:23.495004 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:23.494973    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:23.495136 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:23.495015    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:23.495136 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:23.495032    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:23.496757 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:23.496718    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused"
May 27 23:40:23.496871 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:23.496797    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused"
May 27 23:40:23.496871 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:23.496795    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused"
May 27 23:40:23.496871 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:23.496855    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused"
May 27 23:40:26.494308 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:26.494230    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused"
May 27 23:40:26.494308 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:26.494277    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused"
May 27 23:40:26.494308 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:26.494246    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused"
May 27 23:40:26.494976 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:26.494246    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused"
May 27 23:40:29.494405 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.494361    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:29.494837 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.494570    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:29.494925 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.494912    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:29.494967 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.494955    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:29.494967 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.494965    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:29.495187 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.495163    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:29.495444 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.495426    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:29.495595 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.495469    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:29.495595 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.495496    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:29.499102 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:29.499088    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:30.350460 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:30.350425    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:40:30.353470 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:40:30.353449    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:41:09.663048 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:09.663012    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58dd6bbb87-dxzq5"]
May 27 23:41:32.885717 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:32.885678    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"]
May 27 23:41:32.886163 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:32.886111    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="registry" containerID="cri-o://474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d" gracePeriod=30
May 27 23:41:32.886269 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:32.886214    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="offline" containerID="cri-o://2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d" gracePeriod=30
May 27 23:41:32.886426 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:32.886218    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="online" containerID="cri-o://f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5" gracePeriod=30
May 27 23:41:32.886494 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:32.886258    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="ui" containerID="cri-o://4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8" gracePeriod=30
May 27 23:41:33.173569 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.173469    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"]
May 27 23:41:33.177694 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.177671    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.180524 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.180502    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bjk8p\""
May 27 23:41:33.188689 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.188666    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"]
May 27 23:41:33.285027 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.284982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw64q\" (UniqueName: \"kubernetes.io/projected/60310c34-569a-459f-80df-0df0f11bc307-kube-api-access-bw64q\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.285234 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.285035    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-online-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.285234 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.285141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-ui-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.285234 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.285188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60310c34-569a-459f-80df-0df0f11bc307-feast-data\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.285234 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.285227    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-registry-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.285469 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.285272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-offline-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.386445 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.386399    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-online-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.386663 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.386463    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-ui-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.386663 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.386496    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60310c34-569a-459f-80df-0df0f11bc307-feast-data\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.386663 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.386515    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-registry-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.386663 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:41:33.386579    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 27 23:41:33.386663 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:41:33.386656    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-online-tls podName:60310c34-569a-459f-80df-0df0f11bc307 nodeName:}" failed. No retries permitted until 2026-05-27 23:41:33.886633056 +0000 UTC m=+484.544495507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-online-tls") pod "feast-simple-feast-setup-56d797bf78-wjrgm" (UID: "60310c34-569a-459f-80df-0df0f11bc307") : secret "feast-simple-feast-setup-online-tls" not found
May 27 23:41:33.386978 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.386588    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-offline-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.386978 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.386803    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bw64q\" (UniqueName: \"kubernetes.io/projected/60310c34-569a-459f-80df-0df0f11bc307-kube-api-access-bw64q\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.386978 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.386956    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60310c34-569a-459f-80df-0df0f11bc307-feast-data\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.389111 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.389081    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-ui-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.389461 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.389441    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-offline-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.389564 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.389444    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-registry-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.395180 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.395154    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw64q\" (UniqueName: \"kubernetes.io/projected/60310c34-569a-459f-80df-0df0f11bc307-kube-api-access-bw64q\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.569113 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.569079    2562 generic.go:358] "Generic (PLEG): container finished" podID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerID="4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8" exitCode=0
May 27 23:41:33.569277 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.569124    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerDied","Data":"4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8"}
May 27 23:41:33.891951 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.891857    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-online-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:33.894322 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:33.894298    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-online-tls\") pod \"feast-simple-feast-setup-56d797bf78-wjrgm\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") " pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:34.087479 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:34.087439    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:34.224188 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:34.224161    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"]
May 27 23:41:34.224758 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:41:34.224733    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60310c34_569a_459f_80df_0df0f11bc307.slice/crio-98a8c6972c6ca94b047a5fbb94d5a1bca4585fe57233f55b2110e1e7f4557c65 WatchSource:0}: Error finding container 98a8c6972c6ca94b047a5fbb94d5a1bca4585fe57233f55b2110e1e7f4557c65: Status 404 returned error can't find the container with id 98a8c6972c6ca94b047a5fbb94d5a1bca4585fe57233f55b2110e1e7f4557c65
May 27 23:41:34.573500 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:34.573460    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerStarted","Data":"79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460"}
May 27 23:41:34.573500 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:34.573504    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerStarted","Data":"98a8c6972c6ca94b047a5fbb94d5a1bca4585fe57233f55b2110e1e7f4557c65"}
May 27 23:41:34.576125 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:34.576101    2562 generic.go:358] "Generic (PLEG): container finished" podID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerID="f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5" exitCode=0
May 27 23:41:34.576229 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:34.576177    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerDied","Data":"f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5"}
May 27 23:41:34.683211 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:34.683172    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-58dd6bbb87-dxzq5" podUID="e60cc25d-c6f9-442b-a4d6-debcd9e353bf" containerName="console" containerID="cri-o://370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9" gracePeriod=15
May 27 23:41:34.926509 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:34.926488    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58dd6bbb87-dxzq5_e60cc25d-c6f9-442b-a4d6-debcd9e353bf/console/0.log"
May 27 23:41:34.926850 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:34.926565    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:41:35.002061 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002032    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-oauth-config\") pod \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") "
May 27 23:41:35.002212 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002092    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-service-ca\") pod \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") "
May 27 23:41:35.002212 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002128    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-serving-cert\") pod \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") "
May 27 23:41:35.002212 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002161    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-config\") pod \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") "
May 27 23:41:35.002212 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002198    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-oauth-serving-cert\") pod \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") "
May 27 23:41:35.002461 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002234    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-trusted-ca-bundle\") pod \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") "
May 27 23:41:35.002461 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002252    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6hqd\" (UniqueName: \"kubernetes.io/projected/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-kube-api-access-q6hqd\") pod \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\" (UID: \"e60cc25d-c6f9-442b-a4d6-debcd9e353bf\") "
May 27 23:41:35.002689 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002662    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-service-ca" (OuterVolumeSpecName: "service-ca") pod "e60cc25d-c6f9-442b-a4d6-debcd9e353bf" (UID: "e60cc25d-c6f9-442b-a4d6-debcd9e353bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:41:35.002838 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002806    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-config" (OuterVolumeSpecName: "console-config") pod "e60cc25d-c6f9-442b-a4d6-debcd9e353bf" (UID: "e60cc25d-c6f9-442b-a4d6-debcd9e353bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:41:35.002838 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.002815    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e60cc25d-c6f9-442b-a4d6-debcd9e353bf" (UID: "e60cc25d-c6f9-442b-a4d6-debcd9e353bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:41:35.003050 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.003030    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e60cc25d-c6f9-442b-a4d6-debcd9e353bf" (UID: "e60cc25d-c6f9-442b-a4d6-debcd9e353bf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:41:35.004590 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.004564    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e60cc25d-c6f9-442b-a4d6-debcd9e353bf" (UID: "e60cc25d-c6f9-442b-a4d6-debcd9e353bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:41:35.004698 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.004671    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-kube-api-access-q6hqd" (OuterVolumeSpecName: "kube-api-access-q6hqd") pod "e60cc25d-c6f9-442b-a4d6-debcd9e353bf" (UID: "e60cc25d-c6f9-442b-a4d6-debcd9e353bf"). InnerVolumeSpecName "kube-api-access-q6hqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:41:35.004801 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.004783    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e60cc25d-c6f9-442b-a4d6-debcd9e353bf" (UID: "e60cc25d-c6f9-442b-a4d6-debcd9e353bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:41:35.103966 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.103875    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:41:35.103966 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.103906    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-serving-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:41:35.103966 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.103920    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-config\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:41:35.103966 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.103934    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-oauth-serving-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:41:35.103966 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.103945    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-trusted-ca-bundle\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:41:35.103966 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.103959    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hqd\" (UniqueName: \"kubernetes.io/projected/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-kube-api-access-q6hqd\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:41:35.103966 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.103972    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e60cc25d-c6f9-442b-a4d6-debcd9e353bf-console-oauth-config\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:41:35.583846 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.583812    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58dd6bbb87-dxzq5_e60cc25d-c6f9-442b-a4d6-debcd9e353bf/console/0.log"
May 27 23:41:35.584013 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.583857    2562 generic.go:358] "Generic (PLEG): container finished" podID="e60cc25d-c6f9-442b-a4d6-debcd9e353bf" containerID="370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9" exitCode=2
May 27 23:41:35.584013 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.583927    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58dd6bbb87-dxzq5"
May 27 23:41:35.584013 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.583943    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58dd6bbb87-dxzq5" event={"ID":"e60cc25d-c6f9-442b-a4d6-debcd9e353bf","Type":"ContainerDied","Data":"370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9"}
May 27 23:41:35.584013 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.583982    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58dd6bbb87-dxzq5" event={"ID":"e60cc25d-c6f9-442b-a4d6-debcd9e353bf","Type":"ContainerDied","Data":"4c0d3ac53a619a282b13e315719f086022422381b6fd9f0a5c72c0b4d49a43d9"}
May 27 23:41:35.584013 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.584000    2562 scope.go:117] "RemoveContainer" containerID="370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9"
May 27 23:41:35.593408 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.593393    2562 scope.go:117] "RemoveContainer" containerID="370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9"
May 27 23:41:35.593819 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:41:35.593799    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9\": container with ID starting with 370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9 not found: ID does not exist" containerID="370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9"
May 27 23:41:35.593912 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.593825    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9"} err="failed to get container status \"370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9\": rpc error: code = NotFound desc = could not find container \"370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9\": container with ID starting with 370cecb0083b9a86e38fbacb42446b5a0af86e5465c7d970337736bae5cd5ba9 not found: ID does not exist"
May 27 23:41:35.606680 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.606650    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58dd6bbb87-dxzq5"]
May 27 23:41:35.612959 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.612930    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58dd6bbb87-dxzq5"]
May 27 23:41:35.943967 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:35.943893    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60cc25d-c6f9-442b-a4d6-debcd9e353bf" path="/var/lib/kubelet/pods/e60cc25d-c6f9-442b-a4d6-debcd9e353bf/volumes"
May 27 23:41:39.495850 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:39.495810    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused"
May 27 23:41:39.599103 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:39.599073    2562 generic.go:358] "Generic (PLEG): container finished" podID="60310c34-569a-459f-80df-0df0f11bc307" containerID="79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460" exitCode=0
May 27 23:41:39.599266 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:39.599148    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerDied","Data":"79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460"}
May 27 23:41:40.350730 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:40.350680    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused"
May 27 23:41:40.604030 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:40.603934    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerStarted","Data":"5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc"}
May 27 23:41:45.621842 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:45.621804    2562 generic.go:358] "Generic (PLEG): container finished" podID="60310c34-569a-459f-80df-0df0f11bc307" containerID="5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc" exitCode=0
May 27 23:41:45.622234 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:45.621874    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerDied","Data":"5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc"}
May 27 23:41:46.630630 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:46.630591    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerStarted","Data":"f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43"}
May 27 23:41:46.631126 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:46.630642    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerStarted","Data":"c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40"}
May 27 23:41:46.631126 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:46.630658    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerStarted","Data":"d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058"}
May 27 23:41:46.631126 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:46.630673    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerStarted","Data":"ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4"}
May 27 23:41:46.659352 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:46.659291    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podStartSLOduration=13.659267776 podStartE2EDuration="13.659267776s" podCreationTimestamp="2026-05-27 23:41:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:41:46.656170137 +0000 UTC m=+497.314032632" watchObservedRunningTime="2026-05-27 23:41:46.659267776 +0000 UTC m=+497.317130253"
May 27 23:41:49.088071 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:49.087977    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:49.088071 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:49.088033    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:49.088071 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:49.088045    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:49.088071 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:49.088057    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:49.091126 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:49.091079    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 27 23:41:49.091314 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:49.091174    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused"
May 27 23:41:49.091450 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:49.091260    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused"
May 27 23:41:49.091513 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:49.091496    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused"
May 27 23:41:49.495975 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:49.495842    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused"
May 27 23:41:50.351072 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:50.351020    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused"
May 27 23:41:52.088622 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.088593    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.089026 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.088681    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.089026 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.088904    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.089266 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.089241    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.089369 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.089271    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.089369 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.089285    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.089483 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.089401    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.089565 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.089544    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.089882 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.089869    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.093068 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.093048    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.652605 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.652571    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:52.655608 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:52.655589    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:41:59.495706 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:59.495661    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused"
May 27 23:41:59.496120 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:41:59.495801    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:42:00.350706 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:00.350666    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused"
May 27 23:42:00.350892 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:00.350787    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:42:03.533348 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.533324    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:42:03.660337 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.660262    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wdvh\" (UniqueName: \"kubernetes.io/projected/e3f8a371-e095-42cc-a14d-2e4b68af3d29-kube-api-access-9wdvh\") pod \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") "
May 27 23:42:03.660337 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.660323    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-ui-tls\") pod \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") "
May 27 23:42:03.660549 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.660376    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f8a371-e095-42cc-a14d-2e4b68af3d29-feast-data\") pod \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") "
May 27 23:42:03.660549 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.660424    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-offline-tls\") pod \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") "
May 27 23:42:03.660549 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.660478    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-registry-tls\") pod \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") "
May 27 23:42:03.660707 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.660550    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls\") pod \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\" (UID: \"e3f8a371-e095-42cc-a14d-2e4b68af3d29\") "
May 27 23:42:03.661026 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.660993    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f8a371-e095-42cc-a14d-2e4b68af3d29-feast-data" (OuterVolumeSpecName: "feast-data") pod "e3f8a371-e095-42cc-a14d-2e4b68af3d29" (UID: "e3f8a371-e095-42cc-a14d-2e4b68af3d29"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:42:03.662778 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.662740    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e3f8a371-e095-42cc-a14d-2e4b68af3d29" (UID: "e3f8a371-e095-42cc-a14d-2e4b68af3d29"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:42:03.662778 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.662756    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e3f8a371-e095-42cc-a14d-2e4b68af3d29" (UID: "e3f8a371-e095-42cc-a14d-2e4b68af3d29"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:42:03.662914 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.662782    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e3f8a371-e095-42cc-a14d-2e4b68af3d29" (UID: "e3f8a371-e095-42cc-a14d-2e4b68af3d29"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:42:03.662914 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.662885    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls" (OuterVolumeSpecName: "online-tls") pod "e3f8a371-e095-42cc-a14d-2e4b68af3d29" (UID: "e3f8a371-e095-42cc-a14d-2e4b68af3d29"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:42:03.662981 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.662887    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f8a371-e095-42cc-a14d-2e4b68af3d29-kube-api-access-9wdvh" (OuterVolumeSpecName: "kube-api-access-9wdvh") pod "e3f8a371-e095-42cc-a14d-2e4b68af3d29" (UID: "e3f8a371-e095-42cc-a14d-2e4b68af3d29"). InnerVolumeSpecName "kube-api-access-9wdvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:42:03.691736 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.691707    2562 generic.go:358] "Generic (PLEG): container finished" podID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerID="2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d" exitCode=137
May 27 23:42:03.691736 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.691730    2562 generic.go:358] "Generic (PLEG): container finished" podID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerID="474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d" exitCode=137
May 27 23:42:03.691878 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.691774    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"
May 27 23:42:03.691878 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.691786    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerDied","Data":"2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d"}
May 27 23:42:03.691878 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.691825    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerDied","Data":"474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d"}
May 27 23:42:03.691878 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.691835    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph" event={"ID":"e3f8a371-e095-42cc-a14d-2e4b68af3d29","Type":"ContainerDied","Data":"9108c55c076d60dc212e9ce99662c27d89bfbd6e1f6346bc598069b3a4977275"}
May 27 23:42:03.691878 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.691853    2562 scope.go:117] "RemoveContainer" containerID="4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8"
May 27 23:42:03.700713 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.700700    2562 scope.go:117] "RemoveContainer" containerID="2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d"
May 27 23:42:03.708081 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.708070    2562 scope.go:117] "RemoveContainer" containerID="f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5"
May 27 23:42:03.715301 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.715283    2562 scope.go:117] "RemoveContainer" containerID="474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d"
May 27 23:42:03.716200 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.716175    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"]
May 27 23:42:03.720254 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.720229    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-m6fph"]
May 27 23:42:03.722875 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.722859    2562 scope.go:117] "RemoveContainer" containerID="f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad"
May 27 23:42:03.729642 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.729629    2562 scope.go:117] "RemoveContainer" containerID="49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce"
May 27 23:42:03.738624 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.738600    2562 scope.go:117] "RemoveContainer" containerID="4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8"
May 27 23:42:03.739007 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:42:03.738988    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8\": container with ID starting with 4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8 not found: ID does not exist" containerID="4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8"
May 27 23:42:03.739093 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739016    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8"} err="failed to get container status \"4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8\": rpc error: code = NotFound desc = could not find container \"4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8\": container with ID starting with 4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8 not found: ID does not exist"
May 27 23:42:03.739093 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739036    2562 scope.go:117] "RemoveContainer" containerID="2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d"
May 27 23:42:03.739271 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:42:03.739249    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d\": container with ID starting with 2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d not found: ID does not exist" containerID="2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d"
May 27 23:42:03.739322 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739274    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d"} err="failed to get container status \"2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d\": rpc error: code = NotFound desc = could not find container \"2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d\": container with ID starting with 2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d not found: ID does not exist"
May 27 23:42:03.739322 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739286    2562 scope.go:117] "RemoveContainer" containerID="f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5"
May 27 23:42:03.739496 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:42:03.739482    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5\": container with ID starting with f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5 not found: ID does not exist" containerID="f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5"
May 27 23:42:03.739578 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739499    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5"} err="failed to get container status \"f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5\": rpc error: code = NotFound desc = could not find container \"f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5\": container with ID starting with f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5 not found: ID does not exist"
May 27 23:42:03.739578 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739511    2562 scope.go:117] "RemoveContainer" containerID="474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d"
May 27 23:42:03.739737 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:42:03.739722    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d\": container with ID starting with 474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d not found: ID does not exist" containerID="474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d"
May 27 23:42:03.739788 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739738    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d"} err="failed to get container status \"474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d\": rpc error: code = NotFound desc = could not find container \"474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d\": container with ID starting with 474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d not found: ID does not exist"
May 27 23:42:03.739788 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739750    2562 scope.go:117] "RemoveContainer" containerID="f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad"
May 27 23:42:03.739952 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:42:03.739935    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad\": container with ID starting with f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad not found: ID does not exist" containerID="f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad"
May 27 23:42:03.739990 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739957    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad"} err="failed to get container status \"f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad\": rpc error: code = NotFound desc = could not find container \"f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad\": container with ID starting with f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad not found: ID does not exist"
May 27 23:42:03.739990 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.739974    2562 scope.go:117] "RemoveContainer" containerID="49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce"
May 27 23:42:03.740178 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:42:03.740160    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce\": container with ID starting with 49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce not found: ID does not exist" containerID="49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce"
May 27 23:42:03.740216 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.740183    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce"} err="failed to get container status \"49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce\": rpc error: code = NotFound desc = could not find container \"49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce\": container with ID starting with 49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce not found: ID does not exist"
May 27 23:42:03.740216 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.740199    2562 scope.go:117] "RemoveContainer" containerID="4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8"
May 27 23:42:03.740420 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.740401    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8"} err="failed to get container status \"4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8\": rpc error: code = NotFound desc = could not find container \"4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8\": container with ID starting with 4d9ed93d196a6a4d8ffe0891039198492bb35b6e621bd2a45228a7380c216cc8 not found: ID does not exist"
May 27 23:42:03.740456 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.740421    2562 scope.go:117] "RemoveContainer" containerID="2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d"
May 27 23:42:03.740630 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.740608    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d"} err="failed to get container status \"2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d\": rpc error: code = NotFound desc = could not find container \"2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d\": container with ID starting with 2dd289aa403c69cbdeef0266b5f1c7cb75480516a59b31bced3e60cf4309c84d not found: ID does not exist"
May 27 23:42:03.740669 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.740631    2562 scope.go:117] "RemoveContainer" containerID="f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5"
May 27 23:42:03.740837 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.740818    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5"} err="failed to get container status \"f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5\": rpc error: code = NotFound desc = could not find container \"f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5\": container with ID starting with f849ee41f0fcd747943574ff09d1ea9c81eab5ce2aa3c6aae65f8b06b3cac8c5 not found: ID does not exist"
May 27 23:42:03.740901 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.740838    2562 scope.go:117] "RemoveContainer" containerID="474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d"
May 27 23:42:03.741034 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.741018    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d"} err="failed to get container status \"474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d\": rpc error: code = NotFound desc = could not find container \"474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d\": container with ID starting with 474ae0fb292f97127ff08d0e4941f949fc2e60c4d9fd102e106b2508165e627d not found: ID does not exist"
May 27 23:42:03.741081 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.741034    2562 scope.go:117] "RemoveContainer" containerID="f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad"
May 27 23:42:03.741218 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.741203    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad"} err="failed to get container status \"f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad\": rpc error: code = NotFound desc = could not find container \"f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad\": container with ID starting with f9b505c86a9dff46546dd88cda28549f5df042dc7961c5480c9f0491f65251ad not found: ID does not exist"
May 27 23:42:03.741256 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.741218    2562 scope.go:117] "RemoveContainer" containerID="49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce"
May 27 23:42:03.741392 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.741377    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce"} err="failed to get container status \"49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce\": rpc error: code = NotFound desc = could not find container \"49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce\": container with ID starting with 49e643bdbe2f8cd9f9a0122d5950ff9846109b48efbe9697a622622d908751ce not found: ID does not exist"
May 27 23:42:03.761862 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.761842    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wdvh\" (UniqueName: \"kubernetes.io/projected/e3f8a371-e095-42cc-a14d-2e4b68af3d29-kube-api-access-9wdvh\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:42:03.761862 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.761862    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-ui-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:42:03.761971 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.761872    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f8a371-e095-42cc-a14d-2e4b68af3d29-feast-data\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:42:03.761971 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.761881    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-offline-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:42:03.761971 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.761889    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-registry-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:42:03.761971 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.761896    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3f8a371-e095-42cc-a14d-2e4b68af3d29-online-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:42:03.944163 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:03.944074    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" path="/var/lib/kubelet/pods/e3f8a371-e095-42cc-a14d-2e4b68af3d29/volumes"
May 27 23:42:55.008444 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.008403    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"]
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.008921    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="offline"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.008938    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="offline"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.008949    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="ui"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.008955    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="ui"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.008974    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60cc25d-c6f9-442b-a4d6-debcd9e353bf" containerName="console"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.008980    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60cc25d-c6f9-442b-a4d6-debcd9e353bf" containerName="console"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.008988    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="registry"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.008997    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="registry"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009008    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="feast-apply"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009014    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="feast-apply"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009026    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="feast-init"
May 27 23:42:55.009039 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009035    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="feast-init"
May 27 23:42:55.009418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009046    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="online"
May 27 23:42:55.009418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009054    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="online"
May 27 23:42:55.009418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009115    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="offline"
May 27 23:42:55.009418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009126    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="registry"
May 27 23:42:55.009418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009133    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="ui"
May 27 23:42:55.009418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009142    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="e60cc25d-c6f9-442b-a4d6-debcd9e353bf" containerName="console"
May 27 23:42:55.009418 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.009150    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="e3f8a371-e095-42cc-a14d-2e4b68af3d29" containerName="online"
May 27 23:42:55.012675 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.012661    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.015283 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.015261    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 27 23:42:55.015596 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.015574    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 27 23:42:55.015596 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.015588    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-2kp9v\""
May 27 23:42:55.015784 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.015578    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 27 23:42:55.015784 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.015714    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 27 23:42:55.015897 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.015838    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 27 23:42:55.015897 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.015888    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 27 23:42:55.015996 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.015912    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 27 23:42:55.021289 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.021268    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"]
May 27 23:42:55.203137 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.203090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-ui-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.203320 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.203154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ffd1367-e01b-447f-ae13-5f7477a8857c-registry-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.203320 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.203171    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-online-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.203320 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.203219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffd1367-e01b-447f-ae13-5f7477a8857c-feast-data\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.203320 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.203297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l56n\" (UniqueName: \"kubernetes.io/projected/8ffd1367-e01b-447f-ae13-5f7477a8857c-kube-api-access-2l56n\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.203479 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.203354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-offline-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.303909 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.303815    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffd1367-e01b-447f-ae13-5f7477a8857c-feast-data\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.303909 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.303870    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2l56n\" (UniqueName: \"kubernetes.io/projected/8ffd1367-e01b-447f-ae13-5f7477a8857c-kube-api-access-2l56n\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.303909 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.303902    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-offline-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.304159 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.303928    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-ui-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.304159 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.303961    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ffd1367-e01b-447f-ae13-5f7477a8857c-registry-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.304159 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.303981    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-online-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.304303 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.304278    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffd1367-e01b-447f-ae13-5f7477a8857c-feast-data\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.304696 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.304672    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ffd1367-e01b-447f-ae13-5f7477a8857c-registry-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.306639 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.306618    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-ui-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.306740 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.306679    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-online-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.306740 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.306730    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-offline-tls\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.312582 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.312520    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l56n\" (UniqueName: \"kubernetes.io/projected/8ffd1367-e01b-447f-ae13-5f7477a8857c-kube-api-access-2l56n\") pod \"feast-simple-feast-remote-setup-578f96b6c7-s79nx\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.323598 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.323572    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:42:55.465064 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:42:55.465033    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ffd1367_e01b_447f_ae13_5f7477a8857c.slice/crio-1474c33c856fc24c01128e0b243910db9a042b3f9d1c23a94329d1b79f5096ca WatchSource:0}: Error finding container 1474c33c856fc24c01128e0b243910db9a042b3f9d1c23a94329d1b79f5096ca: Status 404 returned error can't find the container with id 1474c33c856fc24c01128e0b243910db9a042b3f9d1c23a94329d1b79f5096ca
May 27 23:42:55.465064 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.465049    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"]
May 27 23:42:55.862770 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.862726    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerStarted","Data":"a0b5ca304a2e20d572479eb81499dc61ec4b09d715392d7fba61f300cf488115"}
May 27 23:42:55.862770 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:42:55.862773    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerStarted","Data":"1474c33c856fc24c01128e0b243910db9a042b3f9d1c23a94329d1b79f5096ca"}
May 27 23:43:00.879782 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:00.879750    2562 generic.go:358] "Generic (PLEG): container finished" podID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerID="a0b5ca304a2e20d572479eb81499dc61ec4b09d715392d7fba61f300cf488115" exitCode=0
May 27 23:43:00.880259 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:00.879817    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerDied","Data":"a0b5ca304a2e20d572479eb81499dc61ec4b09d715392d7fba61f300cf488115"}
May 27 23:43:01.884681 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:01.884643    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerStarted","Data":"e88b0802d82ab5ad06cfc9f06effcee49b80b9e98ca0b0acb3991d97eeaa681f"}
May 27 23:43:06.902900 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:06.902801    2562 generic.go:358] "Generic (PLEG): container finished" podID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerID="e88b0802d82ab5ad06cfc9f06effcee49b80b9e98ca0b0acb3991d97eeaa681f" exitCode=0
May 27 23:43:06.902900 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:06.902884    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerDied","Data":"e88b0802d82ab5ad06cfc9f06effcee49b80b9e98ca0b0acb3991d97eeaa681f"}
May 27 23:43:07.910077 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:07.910039    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerStarted","Data":"d6aa1845d3959001fa48dbd716052374b35ce5f7b5b062fb25ce78eb9f68cc4a"}
May 27 23:43:07.910582 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:07.910090    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerStarted","Data":"56ba855deaaa8a7acfd364fe5a1c2b73a337e603bfa4fdf813482ae0d39769e6"}
May 27 23:43:07.910582 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:07.910105    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerStarted","Data":"8a85477e69d6f7a442db9d550e52cc423b814a2d2e92603cbaea23e51c04f459"}
May 27 23:43:07.937445 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:07.937380    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podStartSLOduration=13.937356567 podStartE2EDuration="13.937356567s" podCreationTimestamp="2026-05-27 23:42:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:43:07.935411634 +0000 UTC m=+578.593274100" watchObservedRunningTime="2026-05-27 23:43:07.937356567 +0000 UTC m=+578.595219044"
May 27 23:43:10.324778 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:10.324736    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:10.324778 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:10.324787    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:10.325224 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:10.324800    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:10.326833 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:10.326793    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused"
May 27 23:43:10.326833 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:10.326818    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused"
May 27 23:43:10.327033 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:10.326831    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 27 23:43:13.325657 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:13.325624    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:13.326009 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:13.325752    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:13.326101 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:13.326078    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:13.326165 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:13.326112    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:13.326459 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:13.326435    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:13.326591 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:13.326577    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:13.329908 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:13.329890    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:13.931227 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:13.931181    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:13.935275 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:13.935252    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:43:29.886871 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:29.886842    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:43:29.886871 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:29.886867    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:43:29.893017 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:29.892994    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:43:29.893175 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:43:29.893000    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:44:16.107290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:16.107258    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"]
May 27 23:44:16.108365 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:16.108310    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="online" containerID="cri-o://8a85477e69d6f7a442db9d550e52cc423b814a2d2e92603cbaea23e51c04f459" gracePeriod=30
May 27 23:44:16.108561 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:16.108359    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="ui" containerID="cri-o://d6aa1845d3959001fa48dbd716052374b35ce5f7b5b062fb25ce78eb9f68cc4a" gracePeriod=30
May 27 23:44:16.108720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:16.108416    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="offline" containerID="cri-o://56ba855deaaa8a7acfd364fe5a1c2b73a337e603bfa4fdf813482ae0d39769e6" gracePeriod=30
May 27 23:44:16.215909 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:16.215877    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"]
May 27 23:44:16.216318 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:16.216266    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="registry" containerID="cri-o://ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4" gracePeriod=30
May 27 23:44:16.216543 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:16.216307    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="online" containerID="cri-o://d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058" gracePeriod=30
May 27 23:44:16.216666 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:16.216302    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="offline" containerID="cri-o://c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40" gracePeriod=30
May 27 23:44:16.216666 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:16.216301    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="ui" containerID="cri-o://f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43" gracePeriod=30
May 27 23:44:17.134143 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:17.134065    2562 generic.go:358] "Generic (PLEG): container finished" podID="60310c34-569a-459f-80df-0df0f11bc307" containerID="f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43" exitCode=0
May 27 23:44:17.134143 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:17.134089    2562 generic.go:358] "Generic (PLEG): container finished" podID="60310c34-569a-459f-80df-0df0f11bc307" containerID="d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058" exitCode=0
May 27 23:44:17.134143 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:17.134134    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerDied","Data":"f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43"}
May 27 23:44:17.134701 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:17.134166    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerDied","Data":"d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058"}
May 27 23:44:17.136383 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:17.136362    2562 generic.go:358] "Generic (PLEG): container finished" podID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerID="d6aa1845d3959001fa48dbd716052374b35ce5f7b5b062fb25ce78eb9f68cc4a" exitCode=0
May 27 23:44:17.136383 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:17.136383    2562 generic.go:358] "Generic (PLEG): container finished" podID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerID="8a85477e69d6f7a442db9d550e52cc423b814a2d2e92603cbaea23e51c04f459" exitCode=0
May 27 23:44:17.136524 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:17.136421    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerDied","Data":"d6aa1845d3959001fa48dbd716052374b35ce5f7b5b062fb25ce78eb9f68cc4a"}
May 27 23:44:17.136524 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:17.136457    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerDied","Data":"8a85477e69d6f7a442db9d550e52cc423b814a2d2e92603cbaea23e51c04f459"}
May 27 23:44:22.089980 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:22.089909    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused"
May 27 23:44:22.653511 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:22.653465    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 27 23:44:23.327061 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:23.327021    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused"
May 27 23:44:23.931841 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:23.931793    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 27 23:44:32.090100 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:32.090051    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused"
May 27 23:44:32.652965 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:32.652924    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 27 23:44:33.326700 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:33.326652    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused"
May 27 23:44:33.931784 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:33.931738    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 27 23:44:42.090242 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:42.090185    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused"
May 27 23:44:42.090736 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:42.090345    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:44:42.653331 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:42.653290    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 27 23:44:42.653522 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:42.653423    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:44:43.327070 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:43.327033    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused"
May 27 23:44:43.327500 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:43.327138    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:44:43.932414 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:43.932367    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 27 23:44:43.932612 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:43.932504    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:44:46.236082 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.236040    2562 generic.go:358] "Generic (PLEG): container finished" podID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerID="56ba855deaaa8a7acfd364fe5a1c2b73a337e603bfa4fdf813482ae0d39769e6" exitCode=137
May 27 23:44:46.236422 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.236111    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerDied","Data":"56ba855deaaa8a7acfd364fe5a1c2b73a337e603bfa4fdf813482ae0d39769e6"}
May 27 23:44:46.768053 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.768031    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:44:46.864491 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.864471    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:44:46.883267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.883215    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l56n\" (UniqueName: \"kubernetes.io/projected/8ffd1367-e01b-447f-ae13-5f7477a8857c-kube-api-access-2l56n\") pod \"8ffd1367-e01b-447f-ae13-5f7477a8857c\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") "
May 27 23:44:46.883267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.883246    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-offline-tls\") pod \"8ffd1367-e01b-447f-ae13-5f7477a8857c\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") "
May 27 23:44:46.883267 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.883264    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffd1367-e01b-447f-ae13-5f7477a8857c-feast-data\") pod \"8ffd1367-e01b-447f-ae13-5f7477a8857c\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") "
May 27 23:44:46.883495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.883313    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ffd1367-e01b-447f-ae13-5f7477a8857c-registry-tls\") pod \"8ffd1367-e01b-447f-ae13-5f7477a8857c\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") "
May 27 23:44:46.883495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.883339    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-ui-tls\") pod \"8ffd1367-e01b-447f-ae13-5f7477a8857c\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") "
May 27 23:44:46.883495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.883379    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-online-tls\") pod \"8ffd1367-e01b-447f-ae13-5f7477a8857c\" (UID: \"8ffd1367-e01b-447f-ae13-5f7477a8857c\") "
May 27 23:44:46.883772 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.883737    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ffd1367-e01b-447f-ae13-5f7477a8857c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8ffd1367-e01b-447f-ae13-5f7477a8857c" (UID: "8ffd1367-e01b-447f-ae13-5f7477a8857c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:44:46.883885 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.883845    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ffd1367-e01b-447f-ae13-5f7477a8857c-feast-data" (OuterVolumeSpecName: "feast-data") pod "8ffd1367-e01b-447f-ae13-5f7477a8857c" (UID: "8ffd1367-e01b-447f-ae13-5f7477a8857c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:44:46.885434 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.885405    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "8ffd1367-e01b-447f-ae13-5f7477a8857c" (UID: "8ffd1367-e01b-447f-ae13-5f7477a8857c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:46.885516 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.885454    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffd1367-e01b-447f-ae13-5f7477a8857c-kube-api-access-2l56n" (OuterVolumeSpecName: "kube-api-access-2l56n") pod "8ffd1367-e01b-447f-ae13-5f7477a8857c" (UID: "8ffd1367-e01b-447f-ae13-5f7477a8857c"). InnerVolumeSpecName "kube-api-access-2l56n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:44:46.885516 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.885462    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "8ffd1367-e01b-447f-ae13-5f7477a8857c" (UID: "8ffd1367-e01b-447f-ae13-5f7477a8857c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:46.885516 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.885481    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-online-tls" (OuterVolumeSpecName: "online-tls") pod "8ffd1367-e01b-447f-ae13-5f7477a8857c" (UID: "8ffd1367-e01b-447f-ae13-5f7477a8857c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:46.984311 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984277    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-registry-tls\") pod \"60310c34-569a-459f-80df-0df0f11bc307\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") "
May 27 23:44:46.984311 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984312    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60310c34-569a-459f-80df-0df0f11bc307-feast-data\") pod \"60310c34-569a-459f-80df-0df0f11bc307\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") "
May 27 23:44:46.984570 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984341    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-online-tls\") pod \"60310c34-569a-459f-80df-0df0f11bc307\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") "
May 27 23:44:46.984570 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984357    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-offline-tls\") pod \"60310c34-569a-459f-80df-0df0f11bc307\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") "
May 27 23:44:46.984570 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984407    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw64q\" (UniqueName: \"kubernetes.io/projected/60310c34-569a-459f-80df-0df0f11bc307-kube-api-access-bw64q\") pod \"60310c34-569a-459f-80df-0df0f11bc307\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") "
May 27 23:44:46.984570 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984439    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-ui-tls\") pod \"60310c34-569a-459f-80df-0df0f11bc307\" (UID: \"60310c34-569a-459f-80df-0df0f11bc307\") "
May 27 23:44:46.984798 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984681    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l56n\" (UniqueName: \"kubernetes.io/projected/8ffd1367-e01b-447f-ae13-5f7477a8857c-kube-api-access-2l56n\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:46.984798 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984704    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-offline-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:46.984798 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984720    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffd1367-e01b-447f-ae13-5f7477a8857c-feast-data\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:46.984798 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984733    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ffd1367-e01b-447f-ae13-5f7477a8857c-registry-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:46.984798 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984746    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-ui-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:46.984798 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984760    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ffd1367-e01b-447f-ae13-5f7477a8857c-online-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:46.985086 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.984917    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60310c34-569a-459f-80df-0df0f11bc307-feast-data" (OuterVolumeSpecName: "feast-data") pod "60310c34-569a-459f-80df-0df0f11bc307" (UID: "60310c34-569a-459f-80df-0df0f11bc307"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:44:46.986695 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.986652    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-online-tls" (OuterVolumeSpecName: "online-tls") pod "60310c34-569a-459f-80df-0df0f11bc307" (UID: "60310c34-569a-459f-80df-0df0f11bc307"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:46.986808 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.986694    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "60310c34-569a-459f-80df-0df0f11bc307" (UID: "60310c34-569a-459f-80df-0df0f11bc307"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:46.986808 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.986752    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60310c34-569a-459f-80df-0df0f11bc307-kube-api-access-bw64q" (OuterVolumeSpecName: "kube-api-access-bw64q") pod "60310c34-569a-459f-80df-0df0f11bc307" (UID: "60310c34-569a-459f-80df-0df0f11bc307"). InnerVolumeSpecName "kube-api-access-bw64q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:44:46.986942 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.986922    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "60310c34-569a-459f-80df-0df0f11bc307" (UID: "60310c34-569a-459f-80df-0df0f11bc307"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:46.986942 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:46.986933    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "60310c34-569a-459f-80df-0df0f11bc307" (UID: "60310c34-569a-459f-80df-0df0f11bc307"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:47.085330 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.085295    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-online-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:47.085330 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.085322    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-offline-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:47.085330 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.085333    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw64q\" (UniqueName: \"kubernetes.io/projected/60310c34-569a-459f-80df-0df0f11bc307-kube-api-access-bw64q\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:47.085330 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.085342    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-ui-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:47.085621 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.085351    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60310c34-569a-459f-80df-0df0f11bc307-registry-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:47.085621 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.085363    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60310c34-569a-459f-80df-0df0f11bc307-feast-data\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:44:47.244310 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.244286    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"
May 27 23:44:47.244310 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.244294    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx" event={"ID":"8ffd1367-e01b-447f-ae13-5f7477a8857c","Type":"ContainerDied","Data":"1474c33c856fc24c01128e0b243910db9a042b3f9d1c23a94329d1b79f5096ca"}
May 27 23:44:47.244804 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.244336    2562 scope.go:117] "RemoveContainer" containerID="d6aa1845d3959001fa48dbd716052374b35ce5f7b5b062fb25ce78eb9f68cc4a"
May 27 23:44:47.246933 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.246912    2562 generic.go:358] "Generic (PLEG): container finished" podID="60310c34-569a-459f-80df-0df0f11bc307" containerID="c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40" exitCode=137
May 27 23:44:47.246933 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.246931    2562 generic.go:358] "Generic (PLEG): container finished" podID="60310c34-569a-459f-80df-0df0f11bc307" containerID="ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4" exitCode=137
May 27 23:44:47.247105 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.246950    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerDied","Data":"c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40"}
May 27 23:44:47.247105 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.246970    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerDied","Data":"ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4"}
May 27 23:44:47.247105 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.246980    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm" event={"ID":"60310c34-569a-459f-80df-0df0f11bc307","Type":"ContainerDied","Data":"98a8c6972c6ca94b047a5fbb94d5a1bca4585fe57233f55b2110e1e7f4557c65"}
May 27 23:44:47.247105 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.246982    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"
May 27 23:44:47.253729 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.253566    2562 scope.go:117] "RemoveContainer" containerID="56ba855deaaa8a7acfd364fe5a1c2b73a337e603bfa4fdf813482ae0d39769e6"
May 27 23:44:47.261302 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.261278    2562 scope.go:117] "RemoveContainer" containerID="8a85477e69d6f7a442db9d550e52cc423b814a2d2e92603cbaea23e51c04f459"
May 27 23:44:47.268970 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.268954    2562 scope.go:117] "RemoveContainer" containerID="e88b0802d82ab5ad06cfc9f06effcee49b80b9e98ca0b0acb3991d97eeaa681f"
May 27 23:44:47.276075 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.276061    2562 scope.go:117] "RemoveContainer" containerID="a0b5ca304a2e20d572479eb81499dc61ec4b09d715392d7fba61f300cf488115"
May 27 23:44:47.283303 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.283288    2562 scope.go:117] "RemoveContainer" containerID="f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43"
May 27 23:44:47.283818 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.283797    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"]
May 27 23:44:47.291387 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.291368    2562 scope.go:117] "RemoveContainer" containerID="c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40"
May 27 23:44:47.292192 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.292167    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-56d797bf78-wjrgm"]
May 27 23:44:47.298292 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.298275    2562 scope.go:117] "RemoveContainer" containerID="d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058"
May 27 23:44:47.302908 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.302887    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"]
May 27 23:44:47.305778 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.305759    2562 scope.go:117] "RemoveContainer" containerID="ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4"
May 27 23:44:47.306399 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.306382    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-578f96b6c7-s79nx"]
May 27 23:44:47.312990 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.312963    2562 scope.go:117] "RemoveContainer" containerID="5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc"
May 27 23:44:47.319611 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.319596    2562 scope.go:117] "RemoveContainer" containerID="79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460"
May 27 23:44:47.326098 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.326075    2562 scope.go:117] "RemoveContainer" containerID="f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43"
May 27 23:44:47.326347 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:44:47.326328    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43\": container with ID starting with f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43 not found: ID does not exist" containerID="f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43"
May 27 23:44:47.326429 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.326354    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43"} err="failed to get container status \"f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43\": rpc error: code = NotFound desc = could not find container \"f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43\": container with ID starting with f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43 not found: ID does not exist"
May 27 23:44:47.326429 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.326375    2562 scope.go:117] "RemoveContainer" containerID="c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40"
May 27 23:44:47.326624 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:44:47.326605    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40\": container with ID starting with c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40 not found: ID does not exist" containerID="c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40"
May 27 23:44:47.326673 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.326630    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40"} err="failed to get container status \"c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40\": rpc error: code = NotFound desc = could not find container \"c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40\": container with ID starting with c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40 not found: ID does not exist"
May 27 23:44:47.326673 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.326644    2562 scope.go:117] "RemoveContainer" containerID="d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058"
May 27 23:44:47.326866 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:44:47.326848    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058\": container with ID starting with d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058 not found: ID does not exist" containerID="d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058"
May 27 23:44:47.326930 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.326875    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058"} err="failed to get container status \"d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058\": rpc error: code = NotFound desc = could not find container \"d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058\": container with ID starting with d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058 not found: ID does not exist"
May 27 23:44:47.326930 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.326899    2562 scope.go:117] "RemoveContainer" containerID="ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4"
May 27 23:44:47.327116 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:44:47.327099    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4\": container with ID starting with ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4 not found: ID does not exist" containerID="ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4"
May 27 23:44:47.327155 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.327120    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4"} err="failed to get container status \"ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4\": rpc error: code = NotFound desc = could not find container \"ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4\": container with ID starting with ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4 not found: ID does not exist"
May 27 23:44:47.327155 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.327132    2562 scope.go:117] "RemoveContainer" containerID="5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc"
May 27 23:44:47.327314 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:44:47.327298    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc\": container with ID starting with 5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc not found: ID does not exist" containerID="5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc"
May 27 23:44:47.327360 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.327328    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc"} err="failed to get container status \"5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc\": rpc error: code = NotFound desc = could not find container \"5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc\": container with ID starting with 5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc not found: ID does not exist"
May 27 23:44:47.327360 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.327346    2562 scope.go:117] "RemoveContainer" containerID="79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460"
May 27 23:44:47.327653 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:44:47.327510    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460\": container with ID starting with 79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460 not found: ID does not exist" containerID="79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460"
May 27 23:44:47.327653 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.327551    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460"} err="failed to get container status \"79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460\": rpc error: code = NotFound desc = could not find container \"79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460\": container with ID starting with 79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460 not found: ID does not exist"
May 27 23:44:47.327653 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.327572    2562 scope.go:117] "RemoveContainer" containerID="f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43"
May 27 23:44:47.327828 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.327777    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43"} err="failed to get container status \"f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43\": rpc error: code = NotFound desc = could not find container \"f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43\": container with ID starting with f040df3d9d09a074b2a0b98f66ea81d72e778ef20bc48eba732d18d166f1bf43 not found: ID does not exist"
May 27 23:44:47.327828 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.327792    2562 scope.go:117] "RemoveContainer" containerID="c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40"
May 27 23:44:47.328011 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.327990    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40"} err="failed to get container status \"c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40\": rpc error: code = NotFound desc = could not find container \"c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40\": container with ID starting with c0a2adab68fb64b40baa11e6ab106aefd1695fdf420980a0dcbd844992f39b40 not found: ID does not exist"
May 27 23:44:47.328047 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.328012    2562 scope.go:117] "RemoveContainer" containerID="d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058"
May 27 23:44:47.328198 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.328183    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058"} err="failed to get container status \"d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058\": rpc error: code = NotFound desc = could not find container \"d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058\": container with ID starting with d549cc42eaa7eda8ef16493d1b7eb6ab35a15f4dad8bd43b121f3e77165be058 not found: ID does not exist"
May 27 23:44:47.328236 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.328199    2562 scope.go:117] "RemoveContainer" containerID="ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4"
May 27 23:44:47.328384 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.328370    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4"} err="failed to get container status \"ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4\": rpc error: code = NotFound desc = could not find container \"ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4\": container with ID starting with ed714b902c6d9be9c200c309053506a9808e3372f6eeeab74ae1a3c31780a4a4 not found: ID does not exist"
May 27 23:44:47.328421 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.328385    2562 scope.go:117] "RemoveContainer" containerID="5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc"
May 27 23:44:47.328636 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.328615    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc"} err="failed to get container status \"5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc\": rpc error: code = NotFound desc = could not find container \"5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc\": container with ID starting with 5ca3de0df6ffb08f3a0a11d8d14de3dcbc662e6e738961ea7adb7b595503a3bc not found: ID does not exist"
May 27 23:44:47.328686 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.328637    2562 scope.go:117] "RemoveContainer" containerID="79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460"
May 27 23:44:47.328826 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.328810    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460"} err="failed to get container status \"79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460\": rpc error: code = NotFound desc = could not find container \"79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460\": container with ID starting with 79553b33ea5fe947f6ac2fd6906ec3132b9d67fc6707b4ebfa2d212d95020460 not found: ID does not exist"
May 27 23:44:47.944709 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.944678    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60310c34-569a-459f-80df-0df0f11bc307" path="/var/lib/kubelet/pods/60310c34-569a-459f-80df-0df0f11bc307/volumes"
May 27 23:44:47.945290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:47.945277    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" path="/var/lib/kubelet/pods/8ffd1367-e01b-447f-ae13-5f7477a8857c/volumes"
May 27 23:44:59.079455 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079420    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-jqb7z"]
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079827    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="feast-apply"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079841    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="feast-apply"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079849    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="feast-init"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079855    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="feast-init"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079866    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="feast-init"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079875    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="feast-init"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079882    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="registry"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079887    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="registry"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079896    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="online"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079902    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="online"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079912    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="online"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079916    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="online"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079922    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="ui"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079926    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="ui"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079938    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="offline"
May 27 23:44:59.079939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079948    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="offline"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079958    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="ui"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079965    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="ui"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079977    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="feast-apply"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079982    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="feast-apply"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.079988    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="offline"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.080010    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="offline"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.080065    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="offline"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.080073    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="ui"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.080079    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="ui"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.080089    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="registry"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.080099    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="60310c34-569a-459f-80df-0df0f11bc307" containerName="online"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.080107    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="online"
May 27 23:44:59.080452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.080114    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8ffd1367-e01b-447f-ae13-5f7477a8857c" containerName="offline"
May 27 23:44:59.085735 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.085708    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:44:59.087363 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.087321    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-jqb7z"]
May 27 23:44:59.088234 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.088214    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 27 23:44:59.088545 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.088253    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-msr2f\""
May 27 23:44:59.088678 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.088313    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 27 23:44:59.089338 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.089317    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 27 23:44:59.194691 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.194659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5cbc12d0-78cd-433d-9f0c-d6138614adb3-postgresdata\") pod \"postgres-7c7656cc6d-jqb7z\" (UID: \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\") " pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:44:59.194858 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.194718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqdrr\" (UniqueName: \"kubernetes.io/projected/5cbc12d0-78cd-433d-9f0c-d6138614adb3-kube-api-access-rqdrr\") pod \"postgres-7c7656cc6d-jqb7z\" (UID: \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\") " pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:44:59.295234 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.295196    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rqdrr\" (UniqueName: \"kubernetes.io/projected/5cbc12d0-78cd-433d-9f0c-d6138614adb3-kube-api-access-rqdrr\") pod \"postgres-7c7656cc6d-jqb7z\" (UID: \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\") " pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:44:59.295426 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.295288    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5cbc12d0-78cd-433d-9f0c-d6138614adb3-postgresdata\") pod \"postgres-7c7656cc6d-jqb7z\" (UID: \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\") " pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:44:59.295710 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.295688    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5cbc12d0-78cd-433d-9f0c-d6138614adb3-postgresdata\") pod \"postgres-7c7656cc6d-jqb7z\" (UID: \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\") " pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:44:59.304293 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.304271    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqdrr\" (UniqueName: \"kubernetes.io/projected/5cbc12d0-78cd-433d-9f0c-d6138614adb3-kube-api-access-rqdrr\") pod \"postgres-7c7656cc6d-jqb7z\" (UID: \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\") " pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:44:59.404219 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.404145    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:44:59.525932 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.525901    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-jqb7z"]
May 27 23:44:59.528817 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:44:59.528786    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cbc12d0_78cd_433d_9f0c_d6138614adb3.slice/crio-9d89ad555bfba0512b8607821d221b1051085e72dc861a899a585d482301983c WatchSource:0}: Error finding container 9d89ad555bfba0512b8607821d221b1051085e72dc861a899a585d482301983c: Status 404 returned error can't find the container with id 9d89ad555bfba0512b8607821d221b1051085e72dc861a899a585d482301983c
May 27 23:44:59.530542 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:44:59.530515    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 23:45:00.293007 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:00.292969    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z" event={"ID":"5cbc12d0-78cd-433d-9f0c-d6138614adb3","Type":"ContainerStarted","Data":"9d89ad555bfba0512b8607821d221b1051085e72dc861a899a585d482301983c"}
May 27 23:45:05.312656 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:05.312623    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z" event={"ID":"5cbc12d0-78cd-433d-9f0c-d6138614adb3","Type":"ContainerStarted","Data":"1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c"}
May 27 23:45:09.802300 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.802244    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z" podStartSLOduration=5.167232149 podStartE2EDuration="10.802226707s" podCreationTimestamp="2026-05-27 23:44:59 +0000 UTC" firstStartedPulling="2026-05-27 23:44:59.530666429 +0000 UTC m=+690.188528880" lastFinishedPulling="2026-05-27 23:45:05.165660972 +0000 UTC m=+695.823523438" observedRunningTime="2026-05-27 23:45:05.333297669 +0000 UTC m=+695.991160141" watchObservedRunningTime="2026-05-27 23:45:09.802226707 +0000 UTC m=+700.460089180"
May 27 23:45:09.803846 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.803827    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"]
May 27 23:45:09.810716 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.810699    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:09.813420 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.813395    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 27 23:45:09.813420 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.813419    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 27 23:45:09.813625 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.813454    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 27 23:45:09.813625 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.813496    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 27 23:45:09.813625 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.813500    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-nj7mj\""
May 27 23:45:09.817626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.817599    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"]
May 27 23:45:09.990687 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.990651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-offline-tls\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:09.990687 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.990684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw8nx\" (UniqueName: \"kubernetes.io/projected/be320953-604c-4cd3-be4e-6eb999b5bcdc-kube-api-access-pw8nx\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:09.990898 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.990764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-online-tls\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:09.990898 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.990832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-registry-tls\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:09.990898 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:09.990865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be320953-604c-4cd3-be4e-6eb999b5bcdc-feast-data\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.092273 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.092182    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-offline-tls\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.092273 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.092218    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8nx\" (UniqueName: \"kubernetes.io/projected/be320953-604c-4cd3-be4e-6eb999b5bcdc-kube-api-access-pw8nx\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.092475 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.092297    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-online-tls\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.092475 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.092359    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-registry-tls\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.092573 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.092511    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be320953-604c-4cd3-be4e-6eb999b5bcdc-feast-data\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.092907 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.092885    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be320953-604c-4cd3-be4e-6eb999b5bcdc-feast-data\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.094801 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.094780    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-online-tls\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.094881 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.094826    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-offline-tls\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.094881 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.094847    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-registry-tls\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.100422 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.100407    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8nx\" (UniqueName: \"kubernetes.io/projected/be320953-604c-4cd3-be4e-6eb999b5bcdc-kube-api-access-pw8nx\") pod \"feast-credit-scoring-f778ccd46-6h98d\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") " pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.121512 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.121484    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:10.250310 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.250276    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"]
May 27 23:45:10.251611 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:45:10.251585    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe320953_604c_4cd3_be4e_6eb999b5bcdc.slice/crio-474bbae9f383cc4cc312eacaad821a999d8399404f38bb06ea4ec48d7d2af2a7 WatchSource:0}: Error finding container 474bbae9f383cc4cc312eacaad821a999d8399404f38bb06ea4ec48d7d2af2a7: Status 404 returned error can't find the container with id 474bbae9f383cc4cc312eacaad821a999d8399404f38bb06ea4ec48d7d2af2a7
May 27 23:45:10.333458 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.333420    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerStarted","Data":"f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8"}
May 27 23:45:10.333612 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:10.333467    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerStarted","Data":"474bbae9f383cc4cc312eacaad821a999d8399404f38bb06ea4ec48d7d2af2a7"}
May 27 23:45:11.338510 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:11.338423    2562 generic.go:358] "Generic (PLEG): container finished" podID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerID="f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8" exitCode=0
May 27 23:45:11.338510 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:11.338475    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerDied","Data":"f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8"}
May 27 23:45:12.343741 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:12.343661    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerStarted","Data":"f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e"}
May 27 23:45:18.372489 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:18.372455    2562 generic.go:358] "Generic (PLEG): container finished" podID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerID="f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e" exitCode=0
May 27 23:45:18.372929 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:18.372555    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerDied","Data":"f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e"}
May 27 23:45:19.379876 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:19.379833    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerStarted","Data":"196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3"}
May 27 23:45:19.380250 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:19.379891    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerStarted","Data":"2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed"}
May 27 23:45:19.380250 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:19.379906    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerStarted","Data":"ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77"}
May 27 23:45:19.405639 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:19.405581    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podStartSLOduration=10.405560271 podStartE2EDuration="10.405560271s" podCreationTimestamp="2026-05-27 23:45:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:45:19.403958889 +0000 UTC m=+710.061821364" watchObservedRunningTime="2026-05-27 23:45:19.405560271 +0000 UTC m=+710.063422747"
May 27 23:45:20.121808 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:20.121769    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:20.121974 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:20.121822    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:20.121974 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:20.121839    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:22.122036 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:22.121996    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:22.122622 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:22.122594    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:22.122622 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:22.122625    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:22.123900 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:22.123854    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused"
May 27 23:45:22.123900 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:22.123886    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 27 23:45:22.124051 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:22.123867    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused"
May 27 23:45:25.122328 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:25.122272    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused"
May 27 23:45:25.123239 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:25.123217    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:25.124141 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:25.124123    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:25.129589 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:25.129571    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:25.404946 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:25.404862    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:28.121826 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:28.121786    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused"
May 27 23:45:31.123111 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:31.123079    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:45:31.123795 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:45:31.123778    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:46:33.680866 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:33.680824    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"]
May 27 23:46:33.681370 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:33.681149    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="registry" containerID="cri-o://ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77" gracePeriod=30
May 27 23:46:33.681370 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:33.681222    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="offline" containerID="cri-o://196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3" gracePeriod=30
May 27 23:46:33.681370 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:33.681242    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="online" containerID="cri-o://2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed" gracePeriod=30
May 27 23:46:33.698679 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:33.698649    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-jqb7z"]
May 27 23:46:33.698844 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:33.698825    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z" podUID="5cbc12d0-78cd-433d-9f0c-d6138614adb3" containerName="postgres" containerID="cri-o://1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c" gracePeriod=30
May 27 23:46:33.942184 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:33.942118    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:46:34.043663 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.043626    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5cbc12d0-78cd-433d-9f0c-d6138614adb3-postgresdata\") pod \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\" (UID: \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\") "
May 27 23:46:34.043851 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.043761    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqdrr\" (UniqueName: \"kubernetes.io/projected/5cbc12d0-78cd-433d-9f0c-d6138614adb3-kube-api-access-rqdrr\") pod \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\" (UID: \"5cbc12d0-78cd-433d-9f0c-d6138614adb3\") "
May 27 23:46:34.045816 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.045781    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbc12d0-78cd-433d-9f0c-d6138614adb3-kube-api-access-rqdrr" (OuterVolumeSpecName: "kube-api-access-rqdrr") pod "5cbc12d0-78cd-433d-9f0c-d6138614adb3" (UID: "5cbc12d0-78cd-433d-9f0c-d6138614adb3"). InnerVolumeSpecName "kube-api-access-rqdrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:46:34.073634 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.073594    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbc12d0-78cd-433d-9f0c-d6138614adb3-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "5cbc12d0-78cd-433d-9f0c-d6138614adb3" (UID: "5cbc12d0-78cd-433d-9f0c-d6138614adb3"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:46:34.144972 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.144934    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqdrr\" (UniqueName: \"kubernetes.io/projected/5cbc12d0-78cd-433d-9f0c-d6138614adb3-kube-api-access-rqdrr\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:46:34.144972 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.144965    2562 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5cbc12d0-78cd-433d-9f0c-d6138614adb3-postgresdata\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:46:34.630581 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.630472    2562 generic.go:358] "Generic (PLEG): container finished" podID="5cbc12d0-78cd-433d-9f0c-d6138614adb3" containerID="1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c" exitCode=0
May 27 23:46:34.630581 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.630522    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z"
May 27 23:46:34.630581 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.630568    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z" event={"ID":"5cbc12d0-78cd-433d-9f0c-d6138614adb3","Type":"ContainerDied","Data":"1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c"}
May 27 23:46:34.630868 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.630613    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-jqb7z" event={"ID":"5cbc12d0-78cd-433d-9f0c-d6138614adb3","Type":"ContainerDied","Data":"9d89ad555bfba0512b8607821d221b1051085e72dc861a899a585d482301983c"}
May 27 23:46:34.630868 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.630633    2562 scope.go:117] "RemoveContainer" containerID="1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c"
May 27 23:46:34.633097 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.633074    2562 generic.go:358] "Generic (PLEG): container finished" podID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerID="2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed" exitCode=0
May 27 23:46:34.633203 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.633129    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerDied","Data":"2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed"}
May 27 23:46:34.639579 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.639561    2562 scope.go:117] "RemoveContainer" containerID="1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c"
May 27 23:46:34.639828 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:46:34.639805    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c\": container with ID starting with 1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c not found: ID does not exist" containerID="1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c"
May 27 23:46:34.639890 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.639836    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c"} err="failed to get container status \"1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c\": rpc error: code = NotFound desc = could not find container \"1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c\": container with ID starting with 1ef7b8f4898f1070ba810f19b6e3fe4cc7e3a53d6fb1cd2b3e3d16e3fda59a7c not found: ID does not exist"
May 27 23:46:34.652105 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.652080    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-jqb7z"]
May 27 23:46:34.657594 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:34.657571    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-jqb7z"]
May 27 23:46:35.401923 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:35.401877    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 27 23:46:35.943904 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:35.943872    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cbc12d0-78cd-433d-9f0c-d6138614adb3" path="/var/lib/kubelet/pods/5cbc12d0-78cd-433d-9f0c-d6138614adb3/volumes"
May 27 23:46:45.402383 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:45.402299    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 27 23:46:55.402359 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:55.402311    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 27 23:46:55.402818 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:46:55.402448    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:47:03.722389 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:03.722355    2562 generic.go:358] "Generic (PLEG): container finished" podID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerID="196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3" exitCode=137
May 27 23:47:03.722751 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:03.722396    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerDied","Data":"196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3"}
May 27 23:47:04.329073 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.329045    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:47:04.420322 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.420280    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-registry-tls\") pod \"be320953-604c-4cd3-be4e-6eb999b5bcdc\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") "
May 27 23:47:04.420520 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.420348    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-online-tls\") pod \"be320953-604c-4cd3-be4e-6eb999b5bcdc\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") "
May 27 23:47:04.420520 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.420367    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw8nx\" (UniqueName: \"kubernetes.io/projected/be320953-604c-4cd3-be4e-6eb999b5bcdc-kube-api-access-pw8nx\") pod \"be320953-604c-4cd3-be4e-6eb999b5bcdc\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") "
May 27 23:47:04.420520 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.420457    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-offline-tls\") pod \"be320953-604c-4cd3-be4e-6eb999b5bcdc\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") "
May 27 23:47:04.420737 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.420557    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be320953-604c-4cd3-be4e-6eb999b5bcdc-feast-data\") pod \"be320953-604c-4cd3-be4e-6eb999b5bcdc\" (UID: \"be320953-604c-4cd3-be4e-6eb999b5bcdc\") "
May 27 23:47:04.422810 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.422768    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be320953-604c-4cd3-be4e-6eb999b5bcdc-kube-api-access-pw8nx" (OuterVolumeSpecName: "kube-api-access-pw8nx") pod "be320953-604c-4cd3-be4e-6eb999b5bcdc" (UID: "be320953-604c-4cd3-be4e-6eb999b5bcdc"). InnerVolumeSpecName "kube-api-access-pw8nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:47:04.422935 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.422823    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "be320953-604c-4cd3-be4e-6eb999b5bcdc" (UID: "be320953-604c-4cd3-be4e-6eb999b5bcdc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:47:04.422935 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.422846    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-online-tls" (OuterVolumeSpecName: "online-tls") pod "be320953-604c-4cd3-be4e-6eb999b5bcdc" (UID: "be320953-604c-4cd3-be4e-6eb999b5bcdc"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:47:04.423052 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.423022    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "be320953-604c-4cd3-be4e-6eb999b5bcdc" (UID: "be320953-604c-4cd3-be4e-6eb999b5bcdc"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:47:04.429263 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.429230    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be320953-604c-4cd3-be4e-6eb999b5bcdc-feast-data" (OuterVolumeSpecName: "feast-data") pod "be320953-604c-4cd3-be4e-6eb999b5bcdc" (UID: "be320953-604c-4cd3-be4e-6eb999b5bcdc"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:47:04.521406 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.521374    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be320953-604c-4cd3-be4e-6eb999b5bcdc-feast-data\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:47:04.521406 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.521401    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-registry-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:47:04.521406 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.521412    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-online-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:47:04.521668 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.521421    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw8nx\" (UniqueName: \"kubernetes.io/projected/be320953-604c-4cd3-be4e-6eb999b5bcdc-kube-api-access-pw8nx\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:47:04.521668 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.521431    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be320953-604c-4cd3-be4e-6eb999b5bcdc-offline-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:47:04.732837 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.732803    2562 generic.go:358] "Generic (PLEG): container finished" podID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerID="ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77" exitCode=137
May 27 23:47:04.733266 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.732884    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerDied","Data":"ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77"}
May 27 23:47:04.733266 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.732912    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"
May 27 23:47:04.733266 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.732933    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f778ccd46-6h98d" event={"ID":"be320953-604c-4cd3-be4e-6eb999b5bcdc","Type":"ContainerDied","Data":"474bbae9f383cc4cc312eacaad821a999d8399404f38bb06ea4ec48d7d2af2a7"}
May 27 23:47:04.733266 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.732954    2562 scope.go:117] "RemoveContainer" containerID="196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3"
May 27 23:47:04.743158 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.743122    2562 scope.go:117] "RemoveContainer" containerID="2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed"
May 27 23:47:04.751172 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.751152    2562 scope.go:117] "RemoveContainer" containerID="ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77"
May 27 23:47:04.758568 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.758511    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"]
May 27 23:47:04.759242 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.759227    2562 scope.go:117] "RemoveContainer" containerID="f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e"
May 27 23:47:04.762886 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.762859    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-f778ccd46-6h98d"]
May 27 23:47:04.767298 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.767278    2562 scope.go:117] "RemoveContainer" containerID="f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8"
May 27 23:47:04.775157 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.775138    2562 scope.go:117] "RemoveContainer" containerID="196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3"
May 27 23:47:04.775485 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:47:04.775465    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3\": container with ID starting with 196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3 not found: ID does not exist" containerID="196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3"
May 27 23:47:04.775566 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.775496    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3"} err="failed to get container status \"196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3\": rpc error: code = NotFound desc = could not find container \"196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3\": container with ID starting with 196f787f1ba39ba73c78ffa4fe699d21ea50e9bce16d9342cfbdd664c72a07c3 not found: ID does not exist"
May 27 23:47:04.775566 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.775517    2562 scope.go:117] "RemoveContainer" containerID="2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed"
May 27 23:47:04.775818 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:47:04.775801    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed\": container with ID starting with 2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed not found: ID does not exist" containerID="2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed"
May 27 23:47:04.775884 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.775827    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed"} err="failed to get container status \"2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed\": rpc error: code = NotFound desc = could not find container \"2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed\": container with ID starting with 2a85f2c5470bf9ec07c3567726b8585c194a6ade705577ec0b71b3f6a1d533ed not found: ID does not exist"
May 27 23:47:04.775884 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.775851    2562 scope.go:117] "RemoveContainer" containerID="ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77"
May 27 23:47:04.776148 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:47:04.776131    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77\": container with ID starting with ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77 not found: ID does not exist" containerID="ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77"
May 27 23:47:04.776188 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.776153    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77"} err="failed to get container status \"ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77\": rpc error: code = NotFound desc = could not find container \"ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77\": container with ID starting with ac99122cea9b4021a86eb5674fb7725596ea5f1723fa100387948f7cc352fa77 not found: ID does not exist"
May 27 23:47:04.776188 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.776168    2562 scope.go:117] "RemoveContainer" containerID="f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e"
May 27 23:47:04.776359 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:47:04.776343    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e\": container with ID starting with f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e not found: ID does not exist" containerID="f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e"
May 27 23:47:04.776420 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.776362    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e"} err="failed to get container status \"f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e\": rpc error: code = NotFound desc = could not find container \"f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e\": container with ID starting with f1ae331904c4e60d9d48341430da38288eb4963f93fe132d5252da6f1c4a745e not found: ID does not exist"
May 27 23:47:04.776420 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.776375    2562 scope.go:117] "RemoveContainer" containerID="f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8"
May 27 23:47:04.776562 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:47:04.776522    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8\": container with ID starting with f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8 not found: ID does not exist" containerID="f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8"
May 27 23:47:04.776606 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:04.776567    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8"} err="failed to get container status \"f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8\": rpc error: code = NotFound desc = could not find container \"f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8\": container with ID starting with f55f1f272c3acc5880109f1c2deb4b91a3ab731f6beae25d86f543719d3fa7a8 not found: ID does not exist"
May 27 23:47:05.943883 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:47:05.943845    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" path="/var/lib/kubelet/pods/be320953-604c-4cd3-be4e-6eb999b5bcdc/volumes"
May 27 23:48:29.915214 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:29.915185    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:48:29.916318 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:29.916295    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:48:29.920168 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:29.920149    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:48:29.921387 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:29.921369    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:48:54.844343 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.844302    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-2xf77"]
May 27 23:48:54.845196 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845112    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="feast-init"
May 27 23:48:54.845196 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845135    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="feast-init"
May 27 23:48:54.845196 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845149    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cbc12d0-78cd-433d-9f0c-d6138614adb3" containerName="postgres"
May 27 23:48:54.845196 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845164    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cbc12d0-78cd-433d-9f0c-d6138614adb3" containerName="postgres"
May 27 23:48:54.845196 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845188    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="feast-apply"
May 27 23:48:54.845196 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845197    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="feast-apply"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845209    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="offline"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845217    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="offline"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845238    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="online"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845246    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="online"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845270    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="registry"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845278    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="registry"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845422    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="5cbc12d0-78cd-433d-9f0c-d6138614adb3" containerName="postgres"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845445    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="registry"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845455    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="offline"
May 27 23:48:54.845523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.845473    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="be320953-604c-4cd3-be4e-6eb999b5bcdc" containerName="online"
May 27 23:48:54.849570 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.849544    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-2xf77"
May 27 23:48:54.852446 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.852427    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 27 23:48:54.852446 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.852438    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-n6lcc\""
May 27 23:48:54.853495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.853473    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 27 23:48:54.893637 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.893601    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-2xf77"]
May 27 23:48:54.970996 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:54.970952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq4c2\" (UniqueName: \"kubernetes.io/projected/2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43-kube-api-access-xq4c2\") pod \"redis-854544c579-2xf77\" (UID: \"2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43\") " pod="test-ns-feast-rest/redis-854544c579-2xf77"
May 27 23:48:55.071604 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:55.071569    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4c2\" (UniqueName: \"kubernetes.io/projected/2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43-kube-api-access-xq4c2\") pod \"redis-854544c579-2xf77\" (UID: \"2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43\") " pod="test-ns-feast-rest/redis-854544c579-2xf77"
May 27 23:48:55.082875 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:55.082841    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4c2\" (UniqueName: \"kubernetes.io/projected/2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43-kube-api-access-xq4c2\") pod \"redis-854544c579-2xf77\" (UID: \"2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43\") " pod="test-ns-feast-rest/redis-854544c579-2xf77"
May 27 23:48:55.170901 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:55.170805    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-2xf77"
May 27 23:48:55.297768 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:55.297736    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-2xf77"]
May 27 23:48:55.299945 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:48:55.299918    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c18a16d_5ea6_4f46_88d9_6f4d2f7dfe43.slice/crio-e9ce5619259657a4c66e4702b8c025c965c4898146cea81a05eac182ef3fa196 WatchSource:0}: Error finding container e9ce5619259657a4c66e4702b8c025c965c4898146cea81a05eac182ef3fa196: Status 404 returned error can't find the container with id e9ce5619259657a4c66e4702b8c025c965c4898146cea81a05eac182ef3fa196
May 27 23:48:56.072576 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:56.072519    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-2xf77" event={"ID":"2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43","Type":"ContainerStarted","Data":"e9ce5619259657a4c66e4702b8c025c965c4898146cea81a05eac182ef3fa196"}
May 27 23:48:57.077495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:57.077462    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-2xf77" event={"ID":"2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43","Type":"ContainerStarted","Data":"33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088"}
May 27 23:48:57.093652 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:48:57.093590    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-2xf77" podStartSLOduration=1.767444835 podStartE2EDuration="3.093570297s" podCreationTimestamp="2026-05-27 23:48:54 +0000 UTC" firstStartedPulling="2026-05-27 23:48:55.30175562 +0000 UTC m=+925.959618079" lastFinishedPulling="2026-05-27 23:48:56.62788109 +0000 UTC m=+927.285743541" observedRunningTime="2026-05-27 23:48:57.09144463 +0000 UTC m=+927.749307105" watchObservedRunningTime="2026-05-27 23:48:57.093570297 +0000 UTC m=+927.751432773"
May 27 23:49:00.272222 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.272189    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"]
May 27 23:49:00.275750 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.275730    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:49:00.278073 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.278050    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 27 23:49:00.281184 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.281155    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"]
May 27 23:49:00.419351 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.419317    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfw27\" (UniqueName: \"kubernetes.io/projected/340d8f31-3294-437e-8a2d-a12673041c73-kube-api-access-vfw27\") pod \"postgres-7c7656cc6d-l2jpm\" (UID: \"340d8f31-3294-437e-8a2d-a12673041c73\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:49:00.419561 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.419387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/340d8f31-3294-437e-8a2d-a12673041c73-postgresdata\") pod \"postgres-7c7656cc6d-l2jpm\" (UID: \"340d8f31-3294-437e-8a2d-a12673041c73\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:49:00.520874 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.520840    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/340d8f31-3294-437e-8a2d-a12673041c73-postgresdata\") pod \"postgres-7c7656cc6d-l2jpm\" (UID: \"340d8f31-3294-437e-8a2d-a12673041c73\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:49:00.521042 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.520912    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vfw27\" (UniqueName: \"kubernetes.io/projected/340d8f31-3294-437e-8a2d-a12673041c73-kube-api-access-vfw27\") pod \"postgres-7c7656cc6d-l2jpm\" (UID: \"340d8f31-3294-437e-8a2d-a12673041c73\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:49:00.521231 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.521212    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/340d8f31-3294-437e-8a2d-a12673041c73-postgresdata\") pod \"postgres-7c7656cc6d-l2jpm\" (UID: \"340d8f31-3294-437e-8a2d-a12673041c73\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:49:00.529797 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.529740    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfw27\" (UniqueName: \"kubernetes.io/projected/340d8f31-3294-437e-8a2d-a12673041c73-kube-api-access-vfw27\") pod \"postgres-7c7656cc6d-l2jpm\" (UID: \"340d8f31-3294-437e-8a2d-a12673041c73\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:49:00.588290 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.588256    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:49:00.711828 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:00.711802    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"]
May 27 23:49:00.714247 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:49:00.714214    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod340d8f31_3294_437e_8a2d_a12673041c73.slice/crio-2bbf6f3bcf0e9c0110559a6631b16a9db4063aad2029b872ef33e69e05da368d WatchSource:0}: Error finding container 2bbf6f3bcf0e9c0110559a6631b16a9db4063aad2029b872ef33e69e05da368d: Status 404 returned error can't find the container with id 2bbf6f3bcf0e9c0110559a6631b16a9db4063aad2029b872ef33e69e05da368d
May 27 23:49:01.097153 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:01.097116    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm" event={"ID":"340d8f31-3294-437e-8a2d-a12673041c73","Type":"ContainerStarted","Data":"b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58"}
May 27 23:49:01.097153 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:01.097154    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm" event={"ID":"340d8f31-3294-437e-8a2d-a12673041c73","Type":"ContainerStarted","Data":"2bbf6f3bcf0e9c0110559a6631b16a9db4063aad2029b872ef33e69e05da368d"}
May 27 23:49:01.117238 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:01.117190    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm" podStartSLOduration=1.117174321 podStartE2EDuration="1.117174321s" podCreationTimestamp="2026-05-27 23:49:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:49:01.115492911 +0000 UTC m=+931.773355386" watchObservedRunningTime="2026-05-27 23:49:01.117174321 +0000 UTC m=+931.775036795"
May 27 23:49:05.961524 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:05.961490    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"]
May 27 23:49:05.968466 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:05.968444    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:05.971318 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:05.971291    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 27 23:49:05.971433 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:05.971410    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 27 23:49:05.971515 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:05.971472    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-v7ss5\""
May 27 23:49:05.971614 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:05.971599    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 27 23:49:05.977118 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:05.977096    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"]
May 27 23:49:06.069523 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.069488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72bab30a-1b74-436d-acee-8c3585723a36-feast-data\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.069717 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.069557    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-online-tls\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.069717 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.069608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-registry-tls\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.069787 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.069710    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdgjh\" (UniqueName: \"kubernetes.io/projected/72bab30a-1b74-436d-acee-8c3585723a36-kube-api-access-jdgjh\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.170814 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.170779    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jdgjh\" (UniqueName: \"kubernetes.io/projected/72bab30a-1b74-436d-acee-8c3585723a36-kube-api-access-jdgjh\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.170987 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.170831    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72bab30a-1b74-436d-acee-8c3585723a36-feast-data\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.170987 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.170856    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-online-tls\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.170987 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.170897    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-registry-tls\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.171159 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:49:06.171004    2562 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found
May 27 23:49:06.171159 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:49:06.171067    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-registry-tls podName:72bab30a-1b74-436d-acee-8c3585723a36 nodeName:}" failed. No retries permitted until 2026-05-27 23:49:06.671051252 +0000 UTC m=+937.328913704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-registry-tls") pod "feast-credit-scoring-fb5675b6c-2n26p" (UID: "72bab30a-1b74-436d-acee-8c3585723a36") : secret "feast-credit-scoring-registry-tls" not found
May 27 23:49:06.171314 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.171292    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72bab30a-1b74-436d-acee-8c3585723a36-feast-data\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.173480 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.173458    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-online-tls\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.182089 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.182055    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdgjh\" (UniqueName: \"kubernetes.io/projected/72bab30a-1b74-436d-acee-8c3585723a36-kube-api-access-jdgjh\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.679565 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.676981    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-registry-tls\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.684036 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.684005    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-registry-tls\") pod \"feast-credit-scoring-fb5675b6c-2n26p\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") " pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:06.878467 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:06.878414    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:07.030663 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:07.010802    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"]
May 27 23:49:07.030663 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:49:07.013600    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bab30a_1b74_436d_acee_8c3585723a36.slice/crio-824c665569f075b44f96ee0d3873ad629678f32a5d76cef32bb1a941af8eb41e WatchSource:0}: Error finding container 824c665569f075b44f96ee0d3873ad629678f32a5d76cef32bb1a941af8eb41e: Status 404 returned error can't find the container with id 824c665569f075b44f96ee0d3873ad629678f32a5d76cef32bb1a941af8eb41e
May 27 23:49:07.117143 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:07.117113    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerStarted","Data":"002e345e2ebf8e9dcf765d6cf1672431a69c8b3475bdaf3972576e23866ac401"}
May 27 23:49:07.117143 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:07.117149    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerStarted","Data":"824c665569f075b44f96ee0d3873ad629678f32a5d76cef32bb1a941af8eb41e"}
May 27 23:49:08.121393 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:08.121352    2562 generic.go:358] "Generic (PLEG): container finished" podID="72bab30a-1b74-436d-acee-8c3585723a36" containerID="002e345e2ebf8e9dcf765d6cf1672431a69c8b3475bdaf3972576e23866ac401" exitCode=0
May 27 23:49:08.121808 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:08.121408    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerDied","Data":"002e345e2ebf8e9dcf765d6cf1672431a69c8b3475bdaf3972576e23866ac401"}
May 27 23:49:09.126218 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:09.126178    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerStarted","Data":"0b1ec4923d7b970347bb7ad50ecac51be27d12160d30f8858298728523ca5c9e"}
May 27 23:49:16.150580 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:16.150546    2562 generic.go:358] "Generic (PLEG): container finished" podID="72bab30a-1b74-436d-acee-8c3585723a36" containerID="0b1ec4923d7b970347bb7ad50ecac51be27d12160d30f8858298728523ca5c9e" exitCode=0
May 27 23:49:16.150970 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:16.150596    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerDied","Data":"0b1ec4923d7b970347bb7ad50ecac51be27d12160d30f8858298728523ca5c9e"}
May 27 23:49:17.156839 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:17.156801    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerStarted","Data":"b9a41e99b907eba47bed177c7dbf1665a16e3b4954b3cc8aa2ced4009b6f6fda"}
May 27 23:49:17.157339 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:17.156848    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerStarted","Data":"e82e3ad22459b6bd194c91d97d9916374d0ff9b88ff580a0e7b0ce633bc5008e"}
May 27 23:49:17.184740 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:17.184688    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podStartSLOduration=12.184672623 podStartE2EDuration="12.184672623s" podCreationTimestamp="2026-05-27 23:49:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:49:17.183246281 +0000 UTC m=+947.841108755" watchObservedRunningTime="2026-05-27 23:49:17.184672623 +0000 UTC m=+947.842535096"
May 27 23:49:18.878947 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:18.878891    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:18.878947 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:18.878938    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:18.880495 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:18.880459    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused"
May 27 23:49:18.880640 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:18.880471    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 27 23:49:21.880361 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:21.880310    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused"
May 27 23:49:22.330295 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:22.330258    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:22.330752 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:22.330725    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:22.333580 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:22.333557    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:24.879041 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:24.878990    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused"
May 27 23:49:26.878788 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:26.878745    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:27.879268 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:27.879239    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:27.879985 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:27.879967    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:49:31.771144 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.771108    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"]
May 27 23:49:31.776055 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.776036    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:31.778939 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.778916    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 27 23:49:31.779073 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.778944    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 27 23:49:31.779073 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.778993    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-hxts4\""
May 27 23:49:31.779073 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.779063    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 27 23:49:31.785438 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.785413    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"]
May 27 23:49:31.901551 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.901493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-registry-tls\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:31.901720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.901557    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00561a61-a723-42b5-9e87-a4cb499dfbac-feast-data\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:31.901720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.901660    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-online-tls\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:31.901720 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:31.901688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfrp\" (UniqueName: \"kubernetes.io/projected/00561a61-a723-42b5-9e87-a4cb499dfbac-kube-api-access-xmfrp\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.002311 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.002278    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-registry-tls\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.002510 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.002323    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00561a61-a723-42b5-9e87-a4cb499dfbac-feast-data\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.002510 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.002371    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-online-tls\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.002510 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.002388    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfrp\" (UniqueName: \"kubernetes.io/projected/00561a61-a723-42b5-9e87-a4cb499dfbac-kube-api-access-xmfrp\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.002870 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.002845    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00561a61-a723-42b5-9e87-a4cb499dfbac-feast-data\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.004824 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.004803    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-registry-tls\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.004900 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.004870    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-online-tls\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.011050 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.011033    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfrp\" (UniqueName: \"kubernetes.io/projected/00561a61-a723-42b5-9e87-a4cb499dfbac-kube-api-access-xmfrp\") pod \"feast-driver-ranking-6d77d4bc4d-xxb72\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") " pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.085853 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.085765    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:32.214845 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:32.214810    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"]
May 27 23:49:32.218051 ip-10-0-130-189 kubenswrapper[2562]: W0527 23:49:32.218027    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00561a61_a723_42b5_9e87_a4cb499dfbac.slice/crio-11fb5c6810d3fa3ccb190a9a5795972d2aac3b9886243d588852fc47a072b596 WatchSource:0}: Error finding container 11fb5c6810d3fa3ccb190a9a5795972d2aac3b9886243d588852fc47a072b596: Status 404 returned error can't find the container with id 11fb5c6810d3fa3ccb190a9a5795972d2aac3b9886243d588852fc47a072b596
May 27 23:49:33.208122 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:33.208089    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerStarted","Data":"302b95957f2823d8cdbe10962ba8c1ce748fd29e3616cd2172b3b5b7a122d3f4"}
May 27 23:49:33.208591 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:33.208129    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerStarted","Data":"11fb5c6810d3fa3ccb190a9a5795972d2aac3b9886243d588852fc47a072b596"}
May 27 23:49:37.222426 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:37.221273    2562 generic.go:358] "Generic (PLEG): container finished" podID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerID="302b95957f2823d8cdbe10962ba8c1ce748fd29e3616cd2172b3b5b7a122d3f4" exitCode=0
May 27 23:49:37.222426 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:37.221379    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerDied","Data":"302b95957f2823d8cdbe10962ba8c1ce748fd29e3616cd2172b3b5b7a122d3f4"}
May 27 23:49:38.226874 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:38.226844    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerStarted","Data":"3c9b45ffaee1683866eeebe4fa68308785f5e74f3dcdcc9192c8ef0bfee6aeab"}
May 27 23:49:44.249265 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:44.249228    2562 generic.go:358] "Generic (PLEG): container finished" podID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerID="3c9b45ffaee1683866eeebe4fa68308785f5e74f3dcdcc9192c8ef0bfee6aeab" exitCode=0
May 27 23:49:44.249645 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:44.249300    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerDied","Data":"3c9b45ffaee1683866eeebe4fa68308785f5e74f3dcdcc9192c8ef0bfee6aeab"}
May 27 23:49:45.255480 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:45.255444    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerStarted","Data":"880578167dc12168e1aba6f4e43a5f8ee98581c268dd5a551f28ec1ba12dd732"}
May 27 23:49:45.255916 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:45.255489    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerStarted","Data":"c48568ec8af8fa85aac346958b2c65f49f2d4741bf82732c8a7d11617bfd258a"}
May 27 23:49:47.086299 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:47.086259    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:47.086299 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:47.086308    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:47.087833 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:47.087798    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused"
May 27 23:49:47.087833 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:47.087819    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 27 23:49:50.086515 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:50.086471    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused"
May 27 23:49:50.370702 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:50.370619    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:50.371062 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:50.371034    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:50.373816 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:50.373791    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:50.394236 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:50.394174    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podStartSLOduration=19.394158072 podStartE2EDuration="19.394158072s" podCreationTimestamp="2026-05-27 23:49:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:49:45.297193649 +0000 UTC m=+975.955056123" watchObservedRunningTime="2026-05-27 23:49:50.394158072 +0000 UTC m=+981.052020547"
May 27 23:49:52.086545 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:52.086489    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:53.086287 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:53.086244    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused"
May 27 23:49:56.087046 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:56.087018    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:49:56.087772 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:49:56.087750    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:51:10.249512 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.249469    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"]
May 27 23:51:10.250052 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.249804    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226" podUID="ededa324-b6a1-4f96-ae99-f517ffd1d815" containerName="manager" containerID="cri-o://c2917ae7b6bc0d0c2791b3f1da1940e4c23c33b1ac78f841bb0e1b0a5d43a07e" gracePeriod=10
May 27 23:51:10.502161 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.501030    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"]
May 27 23:51:10.502161 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.501417    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="registry" containerID="cri-o://e82e3ad22459b6bd194c91d97d9916374d0ff9b88ff580a0e7b0ce633bc5008e" gracePeriod=30
May 27 23:51:10.502161 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.501827    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="online" containerID="cri-o://b9a41e99b907eba47bed177c7dbf1665a16e3b4954b3cc8aa2ced4009b6f6fda" gracePeriod=30
May 27 23:51:10.511626 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.511580    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"]
May 27 23:51:10.511991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.511939    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="registry" containerID="cri-o://c48568ec8af8fa85aac346958b2c65f49f2d4741bf82732c8a7d11617bfd258a" gracePeriod=30
May 27 23:51:10.511991 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.511975    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="online" containerID="cri-o://880578167dc12168e1aba6f4e43a5f8ee98581c268dd5a551f28ec1ba12dd732" gracePeriod=30
May 27 23:51:10.530296 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.530257    2562 generic.go:358] "Generic (PLEG): container finished" podID="ededa324-b6a1-4f96-ae99-f517ffd1d815" containerID="c2917ae7b6bc0d0c2791b3f1da1940e4c23c33b1ac78f841bb0e1b0a5d43a07e" exitCode=0
May 27 23:51:10.530455 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.530337    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226" event={"ID":"ededa324-b6a1-4f96-ae99-f517ffd1d815","Type":"ContainerDied","Data":"c2917ae7b6bc0d0c2791b3f1da1940e4c23c33b1ac78f841bb0e1b0a5d43a07e"}
May 27 23:51:10.551293 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.551271    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"
May 27 23:51:10.703575 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.703505    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfz8n\" (UniqueName: \"kubernetes.io/projected/ededa324-b6a1-4f96-ae99-f517ffd1d815-kube-api-access-wfz8n\") pod \"ededa324-b6a1-4f96-ae99-f517ffd1d815\" (UID: \"ededa324-b6a1-4f96-ae99-f517ffd1d815\") "
May 27 23:51:10.705677 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.705651    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ededa324-b6a1-4f96-ae99-f517ffd1d815-kube-api-access-wfz8n" (OuterVolumeSpecName: "kube-api-access-wfz8n") pod "ededa324-b6a1-4f96-ae99-f517ffd1d815" (UID: "ededa324-b6a1-4f96-ae99-f517ffd1d815"). InnerVolumeSpecName "kube-api-access-wfz8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:51:10.804911 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:10.804822    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfz8n\" (UniqueName: \"kubernetes.io/projected/ededa324-b6a1-4f96-ae99-f517ffd1d815-kube-api-access-wfz8n\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:11.534419 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.534327    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"
May 27 23:51:11.534419 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.534327    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226" event={"ID":"ededa324-b6a1-4f96-ae99-f517ffd1d815","Type":"ContainerDied","Data":"616ce686ca9f9a56dd18860bb7cdc86664896d7cdc59f31ee69fe474de44d036"}
May 27 23:51:11.534952 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.534454    2562 scope.go:117] "RemoveContainer" containerID="c2917ae7b6bc0d0c2791b3f1da1940e4c23c33b1ac78f841bb0e1b0a5d43a07e"
May 27 23:51:11.536693 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.536663    2562 generic.go:358] "Generic (PLEG): container finished" podID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerID="880578167dc12168e1aba6f4e43a5f8ee98581c268dd5a551f28ec1ba12dd732" exitCode=0
May 27 23:51:11.536817 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.536733    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerDied","Data":"880578167dc12168e1aba6f4e43a5f8ee98581c268dd5a551f28ec1ba12dd732"}
May 27 23:51:11.539183 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.539160    2562 generic.go:358] "Generic (PLEG): container finished" podID="72bab30a-1b74-436d-acee-8c3585723a36" containerID="b9a41e99b907eba47bed177c7dbf1665a16e3b4954b3cc8aa2ced4009b6f6fda" exitCode=0
May 27 23:51:11.539283 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.539209    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerDied","Data":"b9a41e99b907eba47bed177c7dbf1665a16e3b4954b3cc8aa2ced4009b6f6fda"}
May 27 23:51:11.556836 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.556809    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"]
May 27 23:51:11.560832 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.560808    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d47bdf4cc-fs226"]
May 27 23:51:11.944728 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:11.944618    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ededa324-b6a1-4f96-ae99-f517ffd1d815" path="/var/lib/kubelet/pods/ededa324-b6a1-4f96-ae99-f517ffd1d815/volumes"
May 27 23:51:12.331260 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:12.331216    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 27 23:51:12.833519 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:12.833486    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"]
May 27 23:51:12.833947 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:12.833709    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm" podUID="340d8f31-3294-437e-8a2d-a12673041c73" containerName="postgres" containerID="cri-o://b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58" gracePeriod=30
May 27 23:51:12.838096 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:12.838069    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-2xf77"]
May 27 23:51:12.838304 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:12.838280    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-2xf77" podUID="2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43" containerName="redis" containerID="cri-o://33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088" gracePeriod=30
May 27 23:51:13.112921 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.112900    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-2xf77"
May 27 23:51:13.116105 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.116088    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:51:13.226943 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.226907    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq4c2\" (UniqueName: \"kubernetes.io/projected/2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43-kube-api-access-xq4c2\") pod \"2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43\" (UID: \"2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43\") "
May 27 23:51:13.226943 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.226948    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/340d8f31-3294-437e-8a2d-a12673041c73-postgresdata\") pod \"340d8f31-3294-437e-8a2d-a12673041c73\" (UID: \"340d8f31-3294-437e-8a2d-a12673041c73\") "
May 27 23:51:13.227200 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.226967    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfw27\" (UniqueName: \"kubernetes.io/projected/340d8f31-3294-437e-8a2d-a12673041c73-kube-api-access-vfw27\") pod \"340d8f31-3294-437e-8a2d-a12673041c73\" (UID: \"340d8f31-3294-437e-8a2d-a12673041c73\") "
May 27 23:51:13.229175 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.229147    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340d8f31-3294-437e-8a2d-a12673041c73-kube-api-access-vfw27" (OuterVolumeSpecName: "kube-api-access-vfw27") pod "340d8f31-3294-437e-8a2d-a12673041c73" (UID: "340d8f31-3294-437e-8a2d-a12673041c73"). InnerVolumeSpecName "kube-api-access-vfw27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:51:13.229285 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.229242    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43-kube-api-access-xq4c2" (OuterVolumeSpecName: "kube-api-access-xq4c2") pod "2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43" (UID: "2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43"). InnerVolumeSpecName "kube-api-access-xq4c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:51:13.257302 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.257263    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340d8f31-3294-437e-8a2d-a12673041c73-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "340d8f31-3294-437e-8a2d-a12673041c73" (UID: "340d8f31-3294-437e-8a2d-a12673041c73"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:51:13.328285 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.328245    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq4c2\" (UniqueName: \"kubernetes.io/projected/2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43-kube-api-access-xq4c2\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:13.328285 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.328281    2562 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/340d8f31-3294-437e-8a2d-a12673041c73-postgresdata\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:13.328285 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.328293    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfw27\" (UniqueName: \"kubernetes.io/projected/340d8f31-3294-437e-8a2d-a12673041c73-kube-api-access-vfw27\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:13.546819 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.546785    2562 generic.go:358] "Generic (PLEG): container finished" podID="2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43" containerID="33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088" exitCode=0
May 27 23:51:13.547002 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.546830    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-2xf77"
May 27 23:51:13.547002 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.546872    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-2xf77" event={"ID":"2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43","Type":"ContainerDied","Data":"33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088"}
May 27 23:51:13.547002 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.546912    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-2xf77" event={"ID":"2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43","Type":"ContainerDied","Data":"e9ce5619259657a4c66e4702b8c025c965c4898146cea81a05eac182ef3fa196"}
May 27 23:51:13.547002 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.546932    2562 scope.go:117] "RemoveContainer" containerID="33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088"
May 27 23:51:13.548244 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.548222    2562 generic.go:358] "Generic (PLEG): container finished" podID="340d8f31-3294-437e-8a2d-a12673041c73" containerID="b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58" exitCode=0
May 27 23:51:13.548325 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.548263    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm" event={"ID":"340d8f31-3294-437e-8a2d-a12673041c73","Type":"ContainerDied","Data":"b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58"}
May 27 23:51:13.548325 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.548272    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"
May 27 23:51:13.548325 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.548282    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-l2jpm" event={"ID":"340d8f31-3294-437e-8a2d-a12673041c73","Type":"ContainerDied","Data":"2bbf6f3bcf0e9c0110559a6631b16a9db4063aad2029b872ef33e69e05da368d"}
May 27 23:51:13.554902 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.554883    2562 scope.go:117] "RemoveContainer" containerID="33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088"
May 27 23:51:13.555165 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:51:13.555148    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088\": container with ID starting with 33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088 not found: ID does not exist" containerID="33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088"
May 27 23:51:13.555235 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.555173    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088"} err="failed to get container status \"33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088\": rpc error: code = NotFound desc = could not find container \"33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088\": container with ID starting with 33884baa4188762aedd39404900bd49e1b9fb1a547da3673793efdf5d5ba3088 not found: ID does not exist"
May 27 23:51:13.555235 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.555189    2562 scope.go:117] "RemoveContainer" containerID="b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58"
May 27 23:51:13.563271 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.563247    2562 scope.go:117] "RemoveContainer" containerID="b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58"
May 27 23:51:13.563508 ip-10-0-130-189 kubenswrapper[2562]: E0527 23:51:13.563490    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58\": container with ID starting with b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58 not found: ID does not exist" containerID="b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58"
May 27 23:51:13.563581 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.563515    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58"} err="failed to get container status \"b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58\": rpc error: code = NotFound desc = could not find container \"b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58\": container with ID starting with b269cff0aea1c28cc828fd597a777057bf45a5567cec8b83da217e4a2cfaef58 not found: ID does not exist"
May 27 23:51:13.573721 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.573693    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-2xf77"]
May 27 23:51:13.577158 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.577134    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-2xf77"]
May 27 23:51:13.587633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.587605    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"]
May 27 23:51:13.592987 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.592966    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-l2jpm"]
May 27 23:51:13.955058 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.954965    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43" path="/var/lib/kubelet/pods/2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43/volumes"
May 27 23:51:13.955452 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:13.955435    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="340d8f31-3294-437e-8a2d-a12673041c73" path="/var/lib/kubelet/pods/340d8f31-3294-437e-8a2d-a12673041c73/volumes"
May 27 23:51:20.372158 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:20.372109    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 27 23:51:22.331398 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:22.331354    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 27 23:51:30.372184 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:30.372132    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 27 23:51:32.331962 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:32.331915    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 27 23:51:32.332355 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:32.332039    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:51:40.372197 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:40.372153    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 27 23:51:40.372723 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:40.372275    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:51:40.638454 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:40.638377    2562 generic.go:358] "Generic (PLEG): container finished" podID="72bab30a-1b74-436d-acee-8c3585723a36" containerID="e82e3ad22459b6bd194c91d97d9916374d0ff9b88ff580a0e7b0ce633bc5008e" exitCode=137
May 27 23:51:40.638610 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:40.638447    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerDied","Data":"e82e3ad22459b6bd194c91d97d9916374d0ff9b88ff580a0e7b0ce633bc5008e"}
May 27 23:51:40.640378 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:40.640356    2562 generic.go:358] "Generic (PLEG): container finished" podID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerID="c48568ec8af8fa85aac346958b2c65f49f2d4741bf82732c8a7d11617bfd258a" exitCode=137
May 27 23:51:40.640477 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:40.640392    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerDied","Data":"c48568ec8af8fa85aac346958b2c65f49f2d4741bf82732c8a7d11617bfd258a"}
May 27 23:51:41.178647 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.178623    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:51:41.181757 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.181739    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:51:41.269035 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.269006    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-registry-tls\") pod \"00561a61-a723-42b5-9e87-a4cb499dfbac\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") "
May 27 23:51:41.269248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.269052    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-registry-tls\") pod \"72bab30a-1b74-436d-acee-8c3585723a36\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") "
May 27 23:51:41.269248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.269077    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmfrp\" (UniqueName: \"kubernetes.io/projected/00561a61-a723-42b5-9e87-a4cb499dfbac-kube-api-access-xmfrp\") pod \"00561a61-a723-42b5-9e87-a4cb499dfbac\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") "
May 27 23:51:41.269248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.269104    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-online-tls\") pod \"00561a61-a723-42b5-9e87-a4cb499dfbac\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") "
May 27 23:51:41.269248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.269143    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdgjh\" (UniqueName: \"kubernetes.io/projected/72bab30a-1b74-436d-acee-8c3585723a36-kube-api-access-jdgjh\") pod \"72bab30a-1b74-436d-acee-8c3585723a36\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") "
May 27 23:51:41.269248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.269163    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72bab30a-1b74-436d-acee-8c3585723a36-feast-data\") pod \"72bab30a-1b74-436d-acee-8c3585723a36\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") "
May 27 23:51:41.269248 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.269188    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00561a61-a723-42b5-9e87-a4cb499dfbac-feast-data\") pod \"00561a61-a723-42b5-9e87-a4cb499dfbac\" (UID: \"00561a61-a723-42b5-9e87-a4cb499dfbac\") "
May 27 23:51:41.269754 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.269726    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00561a61-a723-42b5-9e87-a4cb499dfbac-feast-data" (OuterVolumeSpecName: "feast-data") pod "00561a61-a723-42b5-9e87-a4cb499dfbac" (UID: "00561a61-a723-42b5-9e87-a4cb499dfbac"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:51:41.269843 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.269222    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-online-tls\") pod \"72bab30a-1b74-436d-acee-8c3585723a36\" (UID: \"72bab30a-1b74-436d-acee-8c3585723a36\") "
May 27 23:51:41.270352 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.270320    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00561a61-a723-42b5-9e87-a4cb499dfbac-feast-data\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:41.271917 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.271848    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-online-tls" (OuterVolumeSpecName: "online-tls") pod "00561a61-a723-42b5-9e87-a4cb499dfbac" (UID: "00561a61-a723-42b5-9e87-a4cb499dfbac"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:51:41.271917 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.271868    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bab30a-1b74-436d-acee-8c3585723a36-kube-api-access-jdgjh" (OuterVolumeSpecName: "kube-api-access-jdgjh") pod "72bab30a-1b74-436d-acee-8c3585723a36" (UID: "72bab30a-1b74-436d-acee-8c3585723a36"). InnerVolumeSpecName "kube-api-access-jdgjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:51:41.271917 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.271868    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00561a61-a723-42b5-9e87-a4cb499dfbac-kube-api-access-xmfrp" (OuterVolumeSpecName: "kube-api-access-xmfrp") pod "00561a61-a723-42b5-9e87-a4cb499dfbac" (UID: "00561a61-a723-42b5-9e87-a4cb499dfbac"). InnerVolumeSpecName "kube-api-access-xmfrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:51:41.271917 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.271871    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "00561a61-a723-42b5-9e87-a4cb499dfbac" (UID: "00561a61-a723-42b5-9e87-a4cb499dfbac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:51:41.272155 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.272135    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "72bab30a-1b74-436d-acee-8c3585723a36" (UID: "72bab30a-1b74-436d-acee-8c3585723a36"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:51:41.272214 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.272184    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-online-tls" (OuterVolumeSpecName: "online-tls") pod "72bab30a-1b74-436d-acee-8c3585723a36" (UID: "72bab30a-1b74-436d-acee-8c3585723a36"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:51:41.278636 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.278616    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bab30a-1b74-436d-acee-8c3585723a36-feast-data" (OuterVolumeSpecName: "feast-data") pod "72bab30a-1b74-436d-acee-8c3585723a36" (UID: "72bab30a-1b74-436d-acee-8c3585723a36"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:51:41.371232 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.371198    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-registry-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:41.371232 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.371229    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-registry-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:41.371232 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.371238    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfrp\" (UniqueName: \"kubernetes.io/projected/00561a61-a723-42b5-9e87-a4cb499dfbac-kube-api-access-xmfrp\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:41.371438 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.371248    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00561a61-a723-42b5-9e87-a4cb499dfbac-online-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:41.371438 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.371259    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdgjh\" (UniqueName: \"kubernetes.io/projected/72bab30a-1b74-436d-acee-8c3585723a36-kube-api-access-jdgjh\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:41.371438 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.371267    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72bab30a-1b74-436d-acee-8c3585723a36-feast-data\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:41.371438 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.371275    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72bab30a-1b74-436d-acee-8c3585723a36-online-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 27 23:51:41.645998 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.645906    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"
May 27 23:51:41.645998 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.645904    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72" event={"ID":"00561a61-a723-42b5-9e87-a4cb499dfbac","Type":"ContainerDied","Data":"11fb5c6810d3fa3ccb190a9a5795972d2aac3b9886243d588852fc47a072b596"}
May 27 23:51:41.646502 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.646031    2562 scope.go:117] "RemoveContainer" containerID="880578167dc12168e1aba6f4e43a5f8ee98581c268dd5a551f28ec1ba12dd732"
May 27 23:51:41.648070 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.648049    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p" event={"ID":"72bab30a-1b74-436d-acee-8c3585723a36","Type":"ContainerDied","Data":"824c665569f075b44f96ee0d3873ad629678f32a5d76cef32bb1a941af8eb41e"}
May 27 23:51:41.648156 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.648097    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"
May 27 23:51:41.654789 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.654742    2562 scope.go:117] "RemoveContainer" containerID="c48568ec8af8fa85aac346958b2c65f49f2d4741bf82732c8a7d11617bfd258a"
May 27 23:51:41.668336 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.668317    2562 scope.go:117] "RemoveContainer" containerID="3c9b45ffaee1683866eeebe4fa68308785f5e74f3dcdcc9192c8ef0bfee6aeab"
May 27 23:51:41.675726 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.675707    2562 scope.go:117] "RemoveContainer" containerID="302b95957f2823d8cdbe10962ba8c1ce748fd29e3616cd2172b3b5b7a122d3f4"
May 27 23:51:41.678633 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.678605    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"]
May 27 23:51:41.682051 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.682032    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fb5675b6c-2n26p"]
May 27 23:51:41.683569 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.683520    2562 scope.go:117] "RemoveContainer" containerID="b9a41e99b907eba47bed177c7dbf1665a16e3b4954b3cc8aa2ced4009b6f6fda"
May 27 23:51:41.690583 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.690566    2562 scope.go:117] "RemoveContainer" containerID="e82e3ad22459b6bd194c91d97d9916374d0ff9b88ff580a0e7b0ce633bc5008e"
May 27 23:51:41.693603 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.693583    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"]
May 27 23:51:41.696708 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.696687    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6d77d4bc4d-xxb72"]
May 27 23:51:41.698307 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.698291    2562 scope.go:117] "RemoveContainer" containerID="0b1ec4923d7b970347bb7ad50ecac51be27d12160d30f8858298728523ca5c9e"
May 27 23:51:41.705176 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.705160    2562 scope.go:117] "RemoveContainer" containerID="002e345e2ebf8e9dcf765d6cf1672431a69c8b3475bdaf3972576e23866ac401"
May 27 23:51:41.944430 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.944349    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" path="/var/lib/kubelet/pods/00561a61-a723-42b5-9e87-a4cb499dfbac/volumes"
May 27 23:51:41.944946 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:51:41.944931    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72bab30a-1b74-436d-acee-8c3585723a36" path="/var/lib/kubelet/pods/72bab30a-1b74-436d-acee-8c3585723a36/volumes"
May 27 23:53:29.938298 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:53:29.938267    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:53:29.940625 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:53:29.940602    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:53:29.944454 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:53:29.944429    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:53:29.946986 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:53:29.946967    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:58:29.962022 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:58:29.961997    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:58:29.964726 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:58:29.964704    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 27 23:58:29.967111 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:58:29.967093    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 27 23:58:29.969823 ip-10-0-130-189 kubenswrapper[2562]: I0527 23:58:29.969806    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 28 00:00:00.184831 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.184798    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29665440-ztdvt"]
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185114    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="registry"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185124    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="registry"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185133    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="online"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185138    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="online"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185143    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="feast-init"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185149    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="feast-init"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185158    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ededa324-b6a1-4f96-ae99-f517ffd1d815" containerName="manager"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185163    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ededa324-b6a1-4f96-ae99-f517ffd1d815" containerName="manager"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185173    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="feast-apply"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185178    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="feast-apply"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185183    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="registry"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185188    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="registry"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185197    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="online"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185202    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="online"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185211    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43" containerName="redis"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185216    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43" containerName="redis"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185220    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="feast-init"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185225    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="feast-init"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185231    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340d8f31-3294-437e-8a2d-a12673041c73" containerName="postgres"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185236    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="340d8f31-3294-437e-8a2d-a12673041c73" containerName="postgres"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185242    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="feast-apply"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185247    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="feast-apply"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185296    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="registry"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185303    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="72bab30a-1b74-436d-acee-8c3585723a36" containerName="online"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185311    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="online"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185317    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="340d8f31-3294-437e-8a2d-a12673041c73" containerName="postgres"
May 28 00:00:00.185312 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185323    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="00561a61-a723-42b5-9e87-a4cb499dfbac" containerName="registry"
May 28 00:00:00.186135 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185331    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="ededa324-b6a1-4f96-ae99-f517ffd1d815" containerName="manager"
May 28 00:00:00.186135 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.185337    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="2c18a16d-5ea6-4f46-88d9-6f4d2f7dfe43" containerName="redis"
May 28 00:00:00.188317 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.188300    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:00:00.195083 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.195059    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-dsddt\""
May 28 00:00:00.195341 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.195327    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
May 28 00:00:00.203109 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.203078    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29665440-ztdvt"]
May 28 00:00:00.322606 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.322574    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-serviceca\") pod \"image-pruner-29665440-ztdvt\" (UID: \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\") " pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:00:00.322792 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.322614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzq9x\" (UniqueName: \"kubernetes.io/projected/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-kube-api-access-xzq9x\") pod \"image-pruner-29665440-ztdvt\" (UID: \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\") " pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:00:00.423556 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.423494    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-serviceca\") pod \"image-pruner-29665440-ztdvt\" (UID: \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\") " pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:00:00.423741 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.423564    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xzq9x\" (UniqueName: \"kubernetes.io/projected/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-kube-api-access-xzq9x\") pod \"image-pruner-29665440-ztdvt\" (UID: \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\") " pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:00:00.424133 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.424112    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-serviceca\") pod \"image-pruner-29665440-ztdvt\" (UID: \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\") " pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:00:00.433441 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.433419    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzq9x\" (UniqueName: \"kubernetes.io/projected/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-kube-api-access-xzq9x\") pod \"image-pruner-29665440-ztdvt\" (UID: \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\") " pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:00:00.522408 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.522368    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:00:00.653560 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.653512    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29665440-ztdvt"]
May 28 00:00:00.656170 ip-10-0-130-189 kubenswrapper[2562]: W0528 00:00:00.656146    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d12640_8608_4acd_8bcb_94fc9ea70b6b.slice/crio-c7cdd38f7e9eed885925bf2989fb3850a43508a7d4fc8fb27f866f4c33472b03 WatchSource:0}: Error finding container c7cdd38f7e9eed885925bf2989fb3850a43508a7d4fc8fb27f866f4c33472b03: Status 404 returned error can't find the container with id c7cdd38f7e9eed885925bf2989fb3850a43508a7d4fc8fb27f866f4c33472b03
May 28 00:00:00.658280 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:00.658265    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:00:01.136069 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:01.136037    2562 generic.go:358] "Generic (PLEG): container finished" podID="b6d12640-8608-4acd-8bcb-94fc9ea70b6b" containerID="972a73338538b8ac175c66f22ea18c3b01eace5582621ea306492ad25c7185c1" exitCode=0
May 28 00:00:01.136248 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:01.136125    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29665440-ztdvt" event={"ID":"b6d12640-8608-4acd-8bcb-94fc9ea70b6b","Type":"ContainerDied","Data":"972a73338538b8ac175c66f22ea18c3b01eace5582621ea306492ad25c7185c1"}
May 28 00:00:01.136248 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:01.136166    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29665440-ztdvt" event={"ID":"b6d12640-8608-4acd-8bcb-94fc9ea70b6b","Type":"ContainerStarted","Data":"c7cdd38f7e9eed885925bf2989fb3850a43508a7d4fc8fb27f866f4c33472b03"}
May 28 00:00:02.256641 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:02.256619    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:00:02.339914 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:02.339882    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzq9x\" (UniqueName: \"kubernetes.io/projected/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-kube-api-access-xzq9x\") pod \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\" (UID: \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\") "
May 28 00:00:02.340105 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:02.339924    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-serviceca\") pod \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\" (UID: \"b6d12640-8608-4acd-8bcb-94fc9ea70b6b\") "
May 28 00:00:02.340449 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:02.340425    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-serviceca" (OuterVolumeSpecName: "serviceca") pod "b6d12640-8608-4acd-8bcb-94fc9ea70b6b" (UID: "b6d12640-8608-4acd-8bcb-94fc9ea70b6b"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:00:02.342115 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:02.342084    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-kube-api-access-xzq9x" (OuterVolumeSpecName: "kube-api-access-xzq9x") pod "b6d12640-8608-4acd-8bcb-94fc9ea70b6b" (UID: "b6d12640-8608-4acd-8bcb-94fc9ea70b6b"). InnerVolumeSpecName "kube-api-access-xzq9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:00:02.441045 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:02.440962    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzq9x\" (UniqueName: \"kubernetes.io/projected/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-kube-api-access-xzq9x\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 28 00:00:02.441045 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:02.440989    2562 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6d12640-8608-4acd-8bcb-94fc9ea70b6b-serviceca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 28 00:00:03.143504 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:03.143465    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29665440-ztdvt" event={"ID":"b6d12640-8608-4acd-8bcb-94fc9ea70b6b","Type":"ContainerDied","Data":"c7cdd38f7e9eed885925bf2989fb3850a43508a7d4fc8fb27f866f4c33472b03"}
May 28 00:00:03.143504 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:03.143501    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7cdd38f7e9eed885925bf2989fb3850a43508a7d4fc8fb27f866f4c33472b03"
May 28 00:00:03.143736 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:00:03.143512    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29665440-ztdvt"
May 28 00:02:04.785190 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.785109    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5894f/must-gather-kkg98"]
May 28 00:02:04.785669 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.785478    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d12640-8608-4acd-8bcb-94fc9ea70b6b" containerName="image-pruner"
May 28 00:02:04.785669 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.785493    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d12640-8608-4acd-8bcb-94fc9ea70b6b" containerName="image-pruner"
May 28 00:02:04.785669 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.785571    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b6d12640-8608-4acd-8bcb-94fc9ea70b6b" containerName="image-pruner"
May 28 00:02:04.788515 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.788496    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:04.791202 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.791176    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5894f\"/\"openshift-service-ca.crt\""
May 28 00:02:04.792259 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.792241    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5894f\"/\"default-dockercfg-gh6mr\""
May 28 00:02:04.792366 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.792277    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5894f\"/\"kube-root-ca.crt\""
May 28 00:02:04.794981 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.794960    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5894f/must-gather-kkg98"]
May 28 00:02:04.877306 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.877265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c758154e-9b12-41fb-93ed-b85281459a70-must-gather-output\") pod \"must-gather-kkg98\" (UID: \"c758154e-9b12-41fb-93ed-b85281459a70\") " pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:04.877306 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.877316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptngj\" (UniqueName: \"kubernetes.io/projected/c758154e-9b12-41fb-93ed-b85281459a70-kube-api-access-ptngj\") pod \"must-gather-kkg98\" (UID: \"c758154e-9b12-41fb-93ed-b85281459a70\") " pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:04.978283 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.978244    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c758154e-9b12-41fb-93ed-b85281459a70-must-gather-output\") pod \"must-gather-kkg98\" (UID: \"c758154e-9b12-41fb-93ed-b85281459a70\") " pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:04.978455 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.978306    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ptngj\" (UniqueName: \"kubernetes.io/projected/c758154e-9b12-41fb-93ed-b85281459a70-kube-api-access-ptngj\") pod \"must-gather-kkg98\" (UID: \"c758154e-9b12-41fb-93ed-b85281459a70\") " pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:04.978614 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.978596    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c758154e-9b12-41fb-93ed-b85281459a70-must-gather-output\") pod \"must-gather-kkg98\" (UID: \"c758154e-9b12-41fb-93ed-b85281459a70\") " pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:04.987033 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:04.987001    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptngj\" (UniqueName: \"kubernetes.io/projected/c758154e-9b12-41fb-93ed-b85281459a70-kube-api-access-ptngj\") pod \"must-gather-kkg98\" (UID: \"c758154e-9b12-41fb-93ed-b85281459a70\") " pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:05.107859 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:05.107761    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:05.238447 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:05.238371    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5894f/must-gather-kkg98"]
May 28 00:02:05.241119 ip-10-0-130-189 kubenswrapper[2562]: W0528 00:02:05.241088    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc758154e_9b12_41fb_93ed_b85281459a70.slice/crio-87e1f438852bb339a814dce68b38be0593ebb64ce8791b97aa2c8833ca540c07 WatchSource:0}: Error finding container 87e1f438852bb339a814dce68b38be0593ebb64ce8791b97aa2c8833ca540c07: Status 404 returned error can't find the container with id 87e1f438852bb339a814dce68b38be0593ebb64ce8791b97aa2c8833ca540c07
May 28 00:02:05.509938 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:05.509905    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5894f/must-gather-kkg98" event={"ID":"c758154e-9b12-41fb-93ed-b85281459a70","Type":"ContainerStarted","Data":"87e1f438852bb339a814dce68b38be0593ebb64ce8791b97aa2c8833ca540c07"}
May 28 00:02:11.535867 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:11.535819    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5894f/must-gather-kkg98" event={"ID":"c758154e-9b12-41fb-93ed-b85281459a70","Type":"ContainerStarted","Data":"3ce50c41bc3f417b62e4198d571a172cca3f3f11a0ea09be6e8f697ace5c4fc6"}
May 28 00:02:12.540775 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:12.540734    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5894f/must-gather-kkg98" event={"ID":"c758154e-9b12-41fb-93ed-b85281459a70","Type":"ContainerStarted","Data":"95bd2270ff2dc2bd7b9954dc42298237231ff881b887c177b1988f1588800a14"}
May 28 00:02:12.558952 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:12.558888    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-5894f/must-gather-kkg98" podStartSLOduration=2.462978278 podStartE2EDuration="8.558869137s" podCreationTimestamp="2026-05-28 00:02:04 +0000 UTC" firstStartedPulling="2026-05-28 00:02:05.242770306 +0000 UTC m=+1715.900632759" lastFinishedPulling="2026-05-28 00:02:11.338661162 +0000 UTC m=+1721.996523618" observedRunningTime="2026-05-28 00:02:12.557731204 +0000 UTC m=+1723.215593692" watchObservedRunningTime="2026-05-28 00:02:12.558869137 +0000 UTC m=+1723.216731611"
May 28 00:02:21.569781 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:21.569747    2562 generic.go:358] "Generic (PLEG): container finished" podID="c758154e-9b12-41fb-93ed-b85281459a70" containerID="3ce50c41bc3f417b62e4198d571a172cca3f3f11a0ea09be6e8f697ace5c4fc6" exitCode=0
May 28 00:02:21.570230 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:21.569819    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5894f/must-gather-kkg98" event={"ID":"c758154e-9b12-41fb-93ed-b85281459a70","Type":"ContainerDied","Data":"3ce50c41bc3f417b62e4198d571a172cca3f3f11a0ea09be6e8f697ace5c4fc6"}
May 28 00:02:21.570230 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:21.570132    2562 scope.go:117] "RemoveContainer" containerID="3ce50c41bc3f417b62e4198d571a172cca3f3f11a0ea09be6e8f697ace5c4fc6"
May 28 00:02:22.068076 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:22.068045    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5894f_must-gather-kkg98_c758154e-9b12-41fb-93ed-b85281459a70/gather/0.log"
May 28 00:02:25.274922 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:25.274890    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vrcnk_e9e31fb0-0d7c-4096-8e90-990753c8c523/global-pull-secret-syncer/0.log"
May 28 00:02:25.371721 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:25.371691    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lzg9b_f493421e-4338-455e-b688-34046f4c4d05/konnectivity-agent/0.log"
May 28 00:02:25.420902 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:25.420870    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-189.ec2.internal_33069df76ded22cb2f23f3545c885186/haproxy/0.log"
May 28 00:02:27.397799 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.397764    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-5894f/must-gather-kkg98"]
May 28 00:02:27.398198 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.397982    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-5894f/must-gather-kkg98" podUID="c758154e-9b12-41fb-93ed-b85281459a70" containerName="copy" containerID="cri-o://95bd2270ff2dc2bd7b9954dc42298237231ff881b887c177b1988f1588800a14" gracePeriod=2
May 28 00:02:27.400418 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.400381    2562 status_manager.go:895] "Failed to get status for pod" podUID="c758154e-9b12-41fb-93ed-b85281459a70" pod="openshift-must-gather-5894f/must-gather-kkg98" err="pods \"must-gather-kkg98\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5894f\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object"
May 28 00:02:27.402194 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.401647    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-5894f/must-gather-kkg98"]
May 28 00:02:27.588858 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.588834    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5894f_must-gather-kkg98_c758154e-9b12-41fb-93ed-b85281459a70/copy/0.log"
May 28 00:02:27.589164 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.589144    2562 generic.go:358] "Generic (PLEG): container finished" podID="c758154e-9b12-41fb-93ed-b85281459a70" containerID="95bd2270ff2dc2bd7b9954dc42298237231ff881b887c177b1988f1588800a14" exitCode=143
May 28 00:02:27.630018 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.629999    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5894f_must-gather-kkg98_c758154e-9b12-41fb-93ed-b85281459a70/copy/0.log"
May 28 00:02:27.630366 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.630350    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:27.632424 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.632403    2562 status_manager.go:895] "Failed to get status for pod" podUID="c758154e-9b12-41fb-93ed-b85281459a70" pod="openshift-must-gather-5894f/must-gather-kkg98" err="pods \"must-gather-kkg98\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5894f\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object"
May 28 00:02:27.787135 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.787099    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptngj\" (UniqueName: \"kubernetes.io/projected/c758154e-9b12-41fb-93ed-b85281459a70-kube-api-access-ptngj\") pod \"c758154e-9b12-41fb-93ed-b85281459a70\" (UID: \"c758154e-9b12-41fb-93ed-b85281459a70\") "
May 28 00:02:27.787319 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.787152    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c758154e-9b12-41fb-93ed-b85281459a70-must-gather-output\") pod \"c758154e-9b12-41fb-93ed-b85281459a70\" (UID: \"c758154e-9b12-41fb-93ed-b85281459a70\") "
May 28 00:02:27.787544 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.787504    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c758154e-9b12-41fb-93ed-b85281459a70-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "c758154e-9b12-41fb-93ed-b85281459a70" (UID: "c758154e-9b12-41fb-93ed-b85281459a70"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:02:27.789287 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.789261    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c758154e-9b12-41fb-93ed-b85281459a70-kube-api-access-ptngj" (OuterVolumeSpecName: "kube-api-access-ptngj") pod "c758154e-9b12-41fb-93ed-b85281459a70" (UID: "c758154e-9b12-41fb-93ed-b85281459a70"). InnerVolumeSpecName "kube-api-access-ptngj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:02:27.887878 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.887826    2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c758154e-9b12-41fb-93ed-b85281459a70-must-gather-output\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 28 00:02:27.887878 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.887871    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptngj\" (UniqueName: \"kubernetes.io/projected/c758154e-9b12-41fb-93ed-b85281459a70-kube-api-access-ptngj\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
May 28 00:02:27.943904 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:27.943876    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c758154e-9b12-41fb-93ed-b85281459a70" path="/var/lib/kubelet/pods/c758154e-9b12-41fb-93ed-b85281459a70/volumes"
May 28 00:02:28.502583 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.502552    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00/alertmanager/0.log"
May 28 00:02:28.533844 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.533815    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00/config-reloader/0.log"
May 28 00:02:28.561443 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.561409    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00/kube-rbac-proxy-web/0.log"
May 28 00:02:28.585640 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.585612    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00/kube-rbac-proxy/0.log"
May 28 00:02:28.593017 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.592998    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5894f_must-gather-kkg98_c758154e-9b12-41fb-93ed-b85281459a70/copy/0.log"
May 28 00:02:28.593322 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.593306    2562 scope.go:117] "RemoveContainer" containerID="95bd2270ff2dc2bd7b9954dc42298237231ff881b887c177b1988f1588800a14"
May 28 00:02:28.593398 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.593343    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5894f/must-gather-kkg98"
May 28 00:02:28.600665 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.600644    2562 scope.go:117] "RemoveContainer" containerID="3ce50c41bc3f417b62e4198d571a172cca3f3f11a0ea09be6e8f697ace5c4fc6"
May 28 00:02:28.613703 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.613682    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00/kube-rbac-proxy-metric/0.log"
May 28 00:02:28.640539 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.640514    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00/prom-label-proxy/0.log"
May 28 00:02:28.666040 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.666002    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3b21d1a6-e11a-4e2f-b65f-44f1a5bb7d00/init-config-reloader/0.log"
May 28 00:02:28.812878 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.812785    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f94fbfbcf-hpgxp_33b19d15-d5cc-41fd-a8fe-26de50041342/metrics-server/0.log"
May 28 00:02:28.842456 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.842423    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-46x9j_f104bebb-24e5-4530-8f45-a68563422250/monitoring-plugin/0.log"
May 28 00:02:28.877882 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.877852    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kqdhj_6f661370-05a6-49dc-a2db-0e20de19d746/node-exporter/0.log"
May 28 00:02:28.902010 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.901986    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kqdhj_6f661370-05a6-49dc-a2db-0e20de19d746/kube-rbac-proxy/0.log"
May 28 00:02:28.929723 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:28.929703    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kqdhj_6f661370-05a6-49dc-a2db-0e20de19d746/init-textfile/0.log"
May 28 00:02:29.133968 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.133886    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-zb5sf_96a40181-7f04-4107-862b-65fb6861574c/kube-rbac-proxy-main/0.log"
May 28 00:02:29.158965 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.158938    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-zb5sf_96a40181-7f04-4107-862b-65fb6861574c/kube-rbac-proxy-self/0.log"
May 28 00:02:29.185014 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.184991    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-zb5sf_96a40181-7f04-4107-862b-65fb6861574c/openshift-state-metrics/0.log"
May 28 00:02:29.405574 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.405478    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-j2sjf_ce9dde2a-6d83-42f6-9499-c6735905726d/prometheus-operator/0.log"
May 28 00:02:29.427469 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.427444    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-j2sjf_ce9dde2a-6d83-42f6-9499-c6735905726d/kube-rbac-proxy/0.log"
May 28 00:02:29.527231 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.527204    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8f56489d8-rkmqn_8a2ae9b1-9e23-4259-bcc0-72b1c8305010/telemeter-client/0.log"
May 28 00:02:29.560201 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.560178    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8f56489d8-rkmqn_8a2ae9b1-9e23-4259-bcc0-72b1c8305010/reload/0.log"
May 28 00:02:29.607248 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.607223    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8f56489d8-rkmqn_8a2ae9b1-9e23-4259-bcc0-72b1c8305010/kube-rbac-proxy/0.log"
May 28 00:02:29.652484 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.652456    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-776d4c97f4-hqmvm_041d9cf1-009b-46e4-a882-d832912ded65/thanos-query/0.log"
May 28 00:02:29.682965 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.682946    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-776d4c97f4-hqmvm_041d9cf1-009b-46e4-a882-d832912ded65/kube-rbac-proxy-web/0.log"
May 28 00:02:29.712266 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.712233    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-776d4c97f4-hqmvm_041d9cf1-009b-46e4-a882-d832912ded65/kube-rbac-proxy/0.log"
May 28 00:02:29.748355 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.748327    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-776d4c97f4-hqmvm_041d9cf1-009b-46e4-a882-d832912ded65/prom-label-proxy/0.log"
May 28 00:02:29.780329 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.780307    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-776d4c97f4-hqmvm_041d9cf1-009b-46e4-a882-d832912ded65/kube-rbac-proxy-rules/0.log"
May 28 00:02:29.827580 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:29.827553    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-776d4c97f4-hqmvm_041d9cf1-009b-46e4-a882-d832912ded65/kube-rbac-proxy-metrics/0.log"
May 28 00:02:30.808610 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:30.808581    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-vznvd_dfee3cde-baf8-4bb9-9a6c-992c1b43e77a/networking-console-plugin/0.log"
May 28 00:02:31.202695 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.202614    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/1.log"
May 28 00:02:31.206999 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.206978    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rgfxl_7359ef3f-6fc9-45cc-bbf6-8fb6a20de37c/console-operator/2.log"
May 28 00:02:31.408980 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.408950    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"]
May 28 00:02:31.409274 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.409262    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c758154e-9b12-41fb-93ed-b85281459a70" containerName="gather"
May 28 00:02:31.409324 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.409276    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c758154e-9b12-41fb-93ed-b85281459a70" containerName="gather"
May 28 00:02:31.409324 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.409289    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c758154e-9b12-41fb-93ed-b85281459a70" containerName="copy"
May 28 00:02:31.409324 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.409294    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c758154e-9b12-41fb-93ed-b85281459a70" containerName="copy"
May 28 00:02:31.409431 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.409357    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="c758154e-9b12-41fb-93ed-b85281459a70" containerName="copy"
May 28 00:02:31.409431 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.409366    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="c758154e-9b12-41fb-93ed-b85281459a70" containerName="gather"
May 28 00:02:31.415638 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.415620    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.418204 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.418172    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kklfx\"/\"openshift-service-ca.crt\""
May 28 00:02:31.418504 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.418485    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kklfx\"/\"default-dockercfg-bnkxh\""
May 28 00:02:31.418610 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.418485    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kklfx\"/\"kube-root-ca.crt\""
May 28 00:02:31.419359 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.419336    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"]
May 28 00:02:31.520740 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.520710    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfctx\" (UniqueName: \"kubernetes.io/projected/8bcdbae3-357c-4118-b818-2c91b21c3ef3-kube-api-access-pfctx\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.520885 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.520750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-proc\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.520885 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.520784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-sys\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.520885 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.520843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-podres\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.521009 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.520905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-lib-modules\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.621476 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.621437    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pfctx\" (UniqueName: \"kubernetes.io/projected/8bcdbae3-357c-4118-b818-2c91b21c3ef3-kube-api-access-pfctx\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.621476 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.621476    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-proc\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.621761 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.621496    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-sys\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.621761 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.621595    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-proc\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.621761 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.621634    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-sys\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.621761 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.621636    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-podres\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.621761 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.621689    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-lib-modules\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.621761 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.621715    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-podres\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.622033 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.621774    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bcdbae3-357c-4118-b818-2c91b21c3ef3-lib-modules\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.629282 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.629261    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfctx\" (UniqueName: \"kubernetes.io/projected/8bcdbae3-357c-4118-b818-2c91b21c3ef3-kube-api-access-pfctx\") pod \"perf-node-gather-daemonset-hf7xb\" (UID: \"8bcdbae3-357c-4118-b818-2c91b21c3ef3\") " pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.726514 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.726475    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:31.849618 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:31.849590    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"]
May 28 00:02:31.852098 ip-10-0-130-189 kubenswrapper[2562]: W0528 00:02:31.852073    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8bcdbae3_357c_4118_b818_2c91b21c3ef3.slice/crio-056c41a14813500ae0b78589a4eac7d9fa1fbe368466cacdace51da44d187d6f WatchSource:0}: Error finding container 056c41a14813500ae0b78589a4eac7d9fa1fbe368466cacdace51da44d187d6f: Status 404 returned error can't find the container with id 056c41a14813500ae0b78589a4eac7d9fa1fbe368466cacdace51da44d187d6f
May 28 00:02:32.024441 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:32.024359    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-mstjh_8c1ef9e8-d3be-4a0b-b38d-b96a11289ee5/volume-data-source-validator/0.log"
May 28 00:02:32.607480 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:32.607444    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb" event={"ID":"8bcdbae3-357c-4118-b818-2c91b21c3ef3","Type":"ContainerStarted","Data":"5600380b3e8ac985524f55f94c2c2d9ae1fa38f80e3689e85afb0f32935d5053"}
May 28 00:02:32.607480 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:32.607483    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb" event={"ID":"8bcdbae3-357c-4118-b818-2c91b21c3ef3","Type":"ContainerStarted","Data":"056c41a14813500ae0b78589a4eac7d9fa1fbe368466cacdace51da44d187d6f"}
May 28 00:02:32.607762 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:32.607556    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:32.623264 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:32.623211    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb" podStartSLOduration=1.623197243 podStartE2EDuration="1.623197243s" podCreationTimestamp="2026-05-28 00:02:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:02:32.622591251 +0000 UTC m=+1743.280453726" watchObservedRunningTime="2026-05-28 00:02:32.623197243 +0000 UTC m=+1743.281059716"
May 28 00:02:32.672768 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:32.672744    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6knhq_9c3a8dde-f0f1-4ebe-88ce-f56cd784f813/dns/0.log"
May 28 00:02:32.696756 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:32.696730    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6knhq_9c3a8dde-f0f1-4ebe-88ce-f56cd784f813/kube-rbac-proxy/0.log"
May 28 00:02:32.849108 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:32.849082    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dbcvm_1f795878-59a3-488e-af3e-9df4e143020a/dns-node-resolver/0.log"
May 28 00:02:33.254747 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:33.254717    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29665440-ztdvt_b6d12640-8608-4acd-8bcb-94fc9ea70b6b/image-pruner/0.log"
May 28 00:02:33.359774 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:33.359744    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j7qvm_4a1534cf-d950-40c0-9a7e-18fc4fc88ee0/node-ca/0.log"
May 28 00:02:34.104514 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:34.104477    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-749c9c75f6-f7z9x_303186fe-da16-4eef-9f9b-6406b1ae5d04/router/0.log"
May 28 00:02:34.433405 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:34.433333    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hwcgw_8accd220-388c-4d68-b5d9-f393b8474848/serve-healthcheck-canary/0.log"
May 28 00:02:34.822201 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:34.822166    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-gtr8n_56d6868c-dba0-46ce-aaa0-78324081e3e4/insights-operator/0.log"
May 28 00:02:34.822703 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:34.822683    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-gtr8n_56d6868c-dba0-46ce-aaa0-78324081e3e4/insights-operator/1.log"
May 28 00:02:34.991145 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:34.991116    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvmmg_feb33324-55b0-422e-85f3-2ad64b45c12d/kube-rbac-proxy/0.log"
May 28 00:02:35.015184 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:35.015161    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvmmg_feb33324-55b0-422e-85f3-2ad64b45c12d/exporter/0.log"
May 28 00:02:35.039078 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:35.039054    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvmmg_feb33324-55b0-422e-85f3-2ad64b45c12d/extractor/0.log"
May 28 00:02:38.620643 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:38.620618    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kklfx/perf-node-gather-daemonset-hf7xb"
May 28 00:02:40.766865 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:40.766842    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgd7k_7ebdbd61-602f-4e9b-95a1-ed43b4d13158/kube-multus-additional-cni-plugins/0.log"
May 28 00:02:40.795516 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:40.795489    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgd7k_7ebdbd61-602f-4e9b-95a1-ed43b4d13158/egress-router-binary-copy/0.log"
May 28 00:02:40.821443 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:40.821417    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgd7k_7ebdbd61-602f-4e9b-95a1-ed43b4d13158/cni-plugins/0.log"
May 28 00:02:40.845350 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:40.845328    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgd7k_7ebdbd61-602f-4e9b-95a1-ed43b4d13158/bond-cni-plugin/0.log"
May 28 00:02:40.875205 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:40.875171    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgd7k_7ebdbd61-602f-4e9b-95a1-ed43b4d13158/routeoverride-cni/0.log"
May 28 00:02:40.900285 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:40.900264    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgd7k_7ebdbd61-602f-4e9b-95a1-ed43b4d13158/whereabouts-cni-bincopy/0.log"
May 28 00:02:40.925408 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:40.925375    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgd7k_7ebdbd61-602f-4e9b-95a1-ed43b4d13158/whereabouts-cni/0.log"
May 28 00:02:41.184648 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:41.184620    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-s5s5h_b71eac2b-1dcb-4cfb-8008-3795f1abab85/kube-multus/0.log"
May 28 00:02:41.208021 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:41.207988    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jd94m_c4abd5dc-33fe-4e84-a433-06f2916dab75/network-metrics-daemon/0.log"
May 28 00:02:41.232087 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:41.232063    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jd94m_c4abd5dc-33fe-4e84-a433-06f2916dab75/kube-rbac-proxy/0.log"
May 28 00:02:42.383558 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:42.383447    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-controller/0.log"
May 28 00:02:42.404817 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:42.404795    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/0.log"
May 28 00:02:42.412537 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:42.412503    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovn-acl-logging/1.log"
May 28 00:02:42.433658 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:42.433633    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/kube-rbac-proxy-node/0.log"
May 28 00:02:42.459547 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:42.459505    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/kube-rbac-proxy-ovn-metrics/0.log"
May 28 00:02:42.481110 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:42.481075    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/northd/0.log"
May 28 00:02:42.507162 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:42.507137    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/nbdb/0.log"
May 28 00:02:42.530807 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:42.530780    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/sbdb/0.log"
May 28 00:02:42.632563 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:42.632115    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khn4g_9e2612fe-61ea-401d-a0b6-b7605e4999ec/ovnkube-controller/0.log"
May 28 00:02:43.911407 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:43.911380    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-fd8x5_bdc00c39-ffab-4200-9e50-bb654d6d6463/check-endpoints/0.log"
May 28 00:02:43.968420 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:43.968388    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qfvtb_c9d94568-264b-4508-9f30-a7bc58643904/network-check-target-container/0.log"
May 28 00:02:44.854992 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:44.854944    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-c6tbg_7c1c6064-ffa6-4522-b6b6-2c6a275fffea/iptables-alerter/0.log"
May 28 00:02:45.538095 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:45.538062    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-cs54f_1d468bad-a43d-40a3-9fdc-9a800365e54c/tuned/0.log"
May 28 00:02:47.281776 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:47.281747    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-2rgz7_61e303c0-6b4f-437a-8f22-17e89a23c62b/cluster-samples-operator/0.log"
May 28 00:02:47.301260 ip-10-0-130-189 kubenswrapper[2562]: I0528 00:02:47.301239    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-2rgz7_61e303c0-6b4f-437a-8f22-17e89a23c62b/cluster-samples-operator-watch/0.log"