May 25 23:32:28.298115 ip-10-0-130-28 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 23:32:28.298132 ip-10-0-130-28 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 23:32:28.298142 ip-10-0-130-28 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 23:32:28.298507 ip-10-0-130-28 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 23:32:39.374023 ip-10-0-130-28 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 25 23:32:39.374046 ip-10-0-130-28 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f6a8a960722a4bf3808b64a2000c1b84 --
May 25 23:35:00.325061 ip-10-0-130-28 systemd[1]: Starting Kubernetes Kubelet...
May 25 23:35:00.786031 ip-10-0-130-28 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 23:35:00.786031 ip-10-0-130-28 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 23:35:00.786031 ip-10-0-130-28 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 23:35:00.786031 ip-10-0-130-28 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 23:35:00.786031 ip-10-0-130-28 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 23:35:00.787969 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.787813    2578 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 23:35:00.791591 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791572    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 23:35:00.791591 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791590    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791594    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791599    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791603    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791606    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791609    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791615    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791619    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791622    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791625    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791628    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791631    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791634    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791637    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791640    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791642    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791645    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791647    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791650    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791653    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 23:35:00.791669 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791656    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791658    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791661    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791663    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791666    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791668    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791671    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791674    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791678    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791682    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791686    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791688    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791691    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791693    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791697    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791700    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791702    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791705    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791707    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 23:35:00.792172 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791710    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791712    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791715    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791717    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791732    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791735    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791737    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791740    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791742    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791745    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791747    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791750    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791753    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791755    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791758    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791761    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791764    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791767    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791769    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791773    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 23:35:00.792685 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791776    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791779    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791781    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791784    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791786    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791789    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791792    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791795    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791797    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791800    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791803    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791805    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791808    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791812    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791814    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791817    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791819    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791822    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791825    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791827    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 23:35:00.793192 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791831    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791833    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791836    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791838    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791841    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.791843    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792258    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792263    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792267    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792269    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792272    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792275    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792278    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792280    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792283    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792285    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792288    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792291    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792293    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792296    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 23:35:00.793692 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792298    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792301    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792304    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792307    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792310    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792314    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792316    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792319    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792322    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792324    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792327    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792329    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792332    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792334    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792337    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792340    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792342    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792345    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792348    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792352    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 23:35:00.794275 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792355    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792357    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792360    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792363    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792365    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792368    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792370    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792373    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792376    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792378    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792380    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792383    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792385    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792388    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792397    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792400    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792403    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792406    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792410    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 23:35:00.794807 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792413    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792416    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792419    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792422    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792425    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792428    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792430    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792433    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792435    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792438    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792441    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792443    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792447    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792449    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792453    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792456    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792458    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792461    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792463    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 23:35:00.795319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792466    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792468    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792471    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792473    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792476    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792479    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792481    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792484    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792486    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792489    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792491    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792494    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792497    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.792499    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792578    2578 flags.go:64] FLAG: --address="0.0.0.0"
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792586    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792593    2578 flags.go:64] FLAG: --anonymous-auth="true"
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792598    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792602    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792605    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792610    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 23:35:00.795809 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792614    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792618    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792621    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792625    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792629    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792632    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792635    2578 flags.go:64] FLAG: --cgroup-root=""
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792638    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792641    2578 flags.go:64] FLAG: --client-ca-file=""
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792644    2578 flags.go:64] FLAG: --cloud-config=""
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792647    2578 flags.go:64] FLAG: --cloud-provider="external"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792650    2578 flags.go:64] FLAG: --cluster-dns="[]"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792654    2578 flags.go:64] FLAG: --cluster-domain=""
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792657    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792661    2578 flags.go:64] FLAG: --config-dir=""
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792663    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792667    2578 flags.go:64] FLAG: --container-log-max-files="5"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792672    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792675    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792678    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792682    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792685    2578 flags.go:64] FLAG: --contention-profiling="false"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792689    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792692    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792695    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 23:35:00.796321 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792698    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792703    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792706    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792709    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792713    2578 flags.go:64] FLAG: --enable-load-reader="false"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792716    2578 flags.go:64] FLAG: --enable-server="true"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792733    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792737    2578 flags.go:64] FLAG: --event-burst="100"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792741    2578 flags.go:64] FLAG: --event-qps="50"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792744    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792747    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792750    2578 flags.go:64] FLAG: --eviction-hard=""
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792754    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792757    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792760    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792763    2578 flags.go:64] FLAG: --eviction-soft=""
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792766    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792769    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792772    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792775    2578 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792778    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792781    2578 flags.go:64] FLAG: --fail-swap-on="true"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792784    2578 flags.go:64] FLAG: --feature-gates=""
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792788    2578 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792791    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 23:35:00.796958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792794    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792797    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792800    2578 flags.go:64] FLAG: --healthz-port="10248"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792803    2578 flags.go:64] FLAG: --help="false"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792807    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-130-28.ec2.internal"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792810    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792813    2578 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792816    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792819    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792824    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792827    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792830    2578 flags.go:64] FLAG: --image-service-endpoint=""
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792833    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792836    2578 flags.go:64] FLAG: --kube-api-burst="100"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792839    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792842    2578 flags.go:64] FLAG: --kube-api-qps="50"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792845    2578 flags.go:64] FLAG: --kube-reserved=""
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792848    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792851    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792854    2578 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792857    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792860    2578 flags.go:64] FLAG: --lock-file=""
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792863    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792866    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 23:35:00.797590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792870    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792876    2578 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792879    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792882    2578 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792885    2578 flags.go:64] FLAG: --logging-format="text"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792888    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792891    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792894    2578 flags.go:64] FLAG: --manifest-url=""
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792897    2578 flags.go:64] FLAG: --manifest-url-header=""
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792902    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792905    2578 flags.go:64] FLAG: --max-open-files="1000000"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792909    2578 flags.go:64] FLAG: --max-pods="110"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792912    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792916    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792919    2578 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792922    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792925    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792928    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792931    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792939    2578 flags.go:64] FLAG: --node-status-max-images="50"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792942    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792945    2578 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792948    2578 flags.go:64] FLAG: --pod-cidr=""
May 25 23:35:00.798183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792951    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792959    2578 flags.go:64] FLAG: --pod-manifest-path=""
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792962    2578 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792968    2578 flags.go:64] FLAG: --pods-per-core="0"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792971    2578 flags.go:64] FLAG: --port="10250"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792974    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792977    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09558b3f759602c55"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792980    2578 flags.go:64] FLAG: --qos-reserved=""
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792983    2578 flags.go:64] FLAG: --read-only-port="10255"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792986    2578 flags.go:64] FLAG: --register-node="true"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792989    2578 flags.go:64] FLAG: --register-schedulable="true"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792992    2578 flags.go:64] FLAG: --register-with-taints=""
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792996    2578 flags.go:64] FLAG: --registry-burst="10"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.792999    2578 flags.go:64] FLAG: --registry-qps="5"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793002    2578 flags.go:64] FLAG: --reserved-cpus=""
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793005    2578 flags.go:64] FLAG: --reserved-memory=""
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793009    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793012    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793015    2578 flags.go:64] FLAG: --rotate-certificates="false"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793018    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793021    2578 flags.go:64] FLAG: --runonce="false"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793024    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793027    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793030    2578 flags.go:64] FLAG: --seccomp-default="false"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793033    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793036    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 23:35:00.798772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793039    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793043    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793046    2578 flags.go:64] FLAG: --storage-driver-password="root"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793049    2578 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793052    2578 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793054    2578 flags.go:64] FLAG: --storage-driver-user="root"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793057    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793060    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793066    2578 flags.go:64] FLAG: --system-cgroups=""
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793070    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793075    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793078    2578 flags.go:64] FLAG: --tls-cert-file=""
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793081    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793086    2578 flags.go:64] FLAG: --tls-min-version=""
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793089    2578 flags.go:64] FLAG: --tls-private-key-file=""
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793092    2578 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793095    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793098    2578 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793101    2578 flags.go:64] FLAG: --v="2"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793106    2578 flags.go:64] FLAG: --version="false"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793110    2578 flags.go:64] FLAG: --vmodule=""
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793114    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.793117    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793211    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 23:35:00.799419 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793215    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793218    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793220    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793223    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793225    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793228    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793231    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793234    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793237    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793239    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793242    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793244    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793247    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793249    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793252    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793255    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793260    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793265    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793268    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793270    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 23:35:00.800015 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793273    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793275    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793278    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793280    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793283    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793285    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793288    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793290    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793293    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793295    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793298    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793301    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793304    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793306    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793309    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793311    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793314    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793316    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793319    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 23:35:00.800580 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793322    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793324    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793327    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793329    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793331    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793334    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793336    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793339    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793341    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793345    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793349    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793351    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793354    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793357    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793359    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793362    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793366    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793369    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793372    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793375    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 23:35:00.801353 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793378    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793381    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793383    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793386    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793389    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793392    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793395    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793397    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793400    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793403    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793405    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793408    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793411    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793414    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793416    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793419    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793421    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793423    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793426    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793428    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 23:35:00.802023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793431    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 23:35:00.802528 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793435    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 23:35:00.802528 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793439    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 23:35:00.802528 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793441    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 23:35:00.802528 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793444    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 23:35:00.802528 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.793448    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 23:35:00.802528 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.794356    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 23:35:00.802907 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.802886    2578 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 23:35:00.802907 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.802907    2578 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 23:35:00.802969 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802963    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 23:35:00.802969 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802969    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802972    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802976    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802979    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802982    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802985    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802987    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802990    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802992    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802996    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.802999    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803001    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803005    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803010    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803013    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803016    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803019    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803022    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803025    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803028    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 23:35:00.803023 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803031    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803034    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803037    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803040    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803043    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803046    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803048    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803051    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803054    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803056    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803059    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803061    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803064    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803066    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803069    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803072    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803074    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803077    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803079    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803081    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 23:35:00.803511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803084    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803086    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803089    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803092    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803094    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803097    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803099    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803102    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803104    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803106    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803109    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803111    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803114    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803117    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803120    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803122    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803125    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803128    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803131    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 23:35:00.804033 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803133    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803136    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803138    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803141    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803143    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803146    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803148    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803151    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803155    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803159    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803161    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803164    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803167    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803169    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803171    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803174    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803177    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803180    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803182    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803185    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 23:35:00.804499 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803188    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803190    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803193    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803196    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803199    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803201    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.803207    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803306    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803311    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803314    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803317    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803319    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803322    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803325    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803328    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 23:35:00.805037 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803331    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803334    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803337    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803339    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803342    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803344    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803347    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803349    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803352    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803355    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803357    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803360    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803363    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803365    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803368    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803371    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803373    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803376    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803379    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803382    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 23:35:00.805425 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803384    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803387    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803389    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803413    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803417    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803421    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803424    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803427    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803430    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803433    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803436    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803438    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803441    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803444    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803446    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803449    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803452    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803454    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803457    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803459    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 23:35:00.806031 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803462    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803464    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803467    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803469    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803472    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803475    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803477    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803480    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803483    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803486    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803490    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803495    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803499    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803502    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803505    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803508    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803511    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803513    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803516    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 23:35:00.806532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803519    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803522    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803524    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803527    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803529    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803532    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803535    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803537    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803540    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803542    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803545    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803547    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803550    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803552    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803555    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803558    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803560    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803563    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 23:35:00.807032 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:00.803565    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 23:35:00.807472 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.803570    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 23:35:00.807472 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.804342    2578 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 23:35:00.807472 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.806945    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 23:35:00.807941 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.807928    2578 server.go:1019] "Starting client certificate rotation"
May 25 23:35:00.808044 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.808025    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 23:35:00.808078 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.808062    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 23:35:00.837257 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.837234    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 23:35:00.838477 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.838458    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 23:35:00.855363 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.855154    2578 log.go:25] "Validated CRI v1 runtime API"
May 25 23:35:00.861678 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.861648    2578 log.go:25] "Validated CRI v1 image API"
May 25 23:35:00.863035 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.863014    2578 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 23:35:00.865495 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.865473    2578 fs.go:135] Filesystem UUIDs: map[78c755eb-b4ac-49aa-ba94-c62ba183ddd7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 82161860-5fc5-4111-8907-8835538e627d:/dev/nvme0n1p3]
May 25 23:35:00.865582 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.865494    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 23:35:00.870178 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.870056    2578 manager.go:217] Machine: {Timestamp:2026-05-25 23:35:00.869535509 +0000 UTC m=+0.424479760 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115572 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d9a39400aba980eced19744babbd1 SystemUUID:ec2d9a39-400a-ba98-0ece-d19744babbd1 BootID:f6a8a960-722a-4bf3-808b-64a2000c1b84 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:57:01:54:ac:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:57:01:54:ac:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:cd:3d:58:1c:90 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 23:35:00.870893 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.870877    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 23:35:00.871862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.871848    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 23:35:00.871968 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.871954    2578 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 23:35:00.873050 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.873025    2578 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 23:35:00.873195 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.873051    2578 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 23:35:00.873240 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.873205    2578 topology_manager.go:138] "Creating topology manager with none policy"
May 25 23:35:00.873240 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.873213    2578 container_manager_linux.go:306] "Creating device plugin manager"
May 25 23:35:00.873240 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.873220    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 23:35:00.874003 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.873992    2578 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 23:35:00.876605 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.876594    2578 state_mem.go:36] "Initialized new in-memory state store"
May 25 23:35:00.876739 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.876715    2578 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 23:35:00.879307 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.879286    2578 kubelet.go:491] "Attempting to sync node with API server"
May 25 23:35:00.879307 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.879315    2578 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 23:35:00.879384 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.879330    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 23:35:00.879384 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.879338    2578 kubelet.go:397] "Adding apiserver pod source"
May 25 23:35:00.879384 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.879347    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 23:35:00.880578 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.880559    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 23:35:00.880657 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.880584    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 23:35:00.883991 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.883971    2578 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 23:35:00.886169 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.886151    2578 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 23:35:00.887044 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887029    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 23:35:00.887128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887049    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 23:35:00.887128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887059    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 23:35:00.887128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887068    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 23:35:00.887128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887080    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 23:35:00.887128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887090    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 23:35:00.887128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887100    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 23:35:00.887128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887109    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 23:35:00.887128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887120    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 23:35:00.887128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887130    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 23:35:00.887450 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887143    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 23:35:00.887450 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.887316    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 23:35:00.888226 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.888214    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 23:35:00.888291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.888229    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 23:35:00.891935 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.891918    2578 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 23:35:00.892033 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.891976    2578 server.go:1295] "Started kubelet"
May 25 23:35:00.892183 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.892118    2578 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 23:35:00.892228 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.892186    2578 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 23:35:00.892228 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.892217    2578 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 23:35:00.892433 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:00.892409    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 23:35:00.892517 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:00.892492    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 23:35:00.892589 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.892555    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 23:35:00.892848 ip-10-0-130-28 systemd[1]: Started Kubernetes Kubelet.
May 25 23:35:00.893448 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.893395    2578 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 23:35:00.895750 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.895736    2578 server.go:317] "Adding debug handlers to kubelet server"
May 25 23:35:00.898289 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:00.897267    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-28.ec2.internal.18b2f31b31eb3234  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-28.ec2.internal,UID:ip-10-0-130-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-28.ec2.internal,},FirstTimestamp:2026-05-25 23:35:00.891931188 +0000 UTC m=+0.446875446,LastTimestamp:2026-05-25 23:35:00.891931188 +0000 UTC m=+0.446875446,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-28.ec2.internal,}"
May 25 23:35:00.899096 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.899077    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 23:35:00.899715 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.899696    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 23:35:00.899857 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.899842    2578 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 23:35:00.899857 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.899857    2578 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 23:35:00.900004 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.899987    2578 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 23:35:00.900072 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.900050    2578 reconstruct.go:97] "Volume reconstruction finished"
May 25 23:35:00.900072 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.900062    2578 reconciler.go:26] "Reconciler: start to sync state"
May 25 23:35:00.902164 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.902027    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vqx64"
May 25 23:35:00.902505 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:00.902471    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:00.904508 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:00.904473    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 23:35:00.904508 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.904477    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 23:35:00.904654 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.904519    2578 factory.go:55] Registering systemd factory
May 25 23:35:00.904654 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.904532    2578 factory.go:223] Registration of the systemd container factory successfully
May 25 23:35:00.904654 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:00.904625    2578 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 23:35:00.905387 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.905133    2578 factory.go:153] Registering CRI-O factory
May 25 23:35:00.905387 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.905151    2578 factory.go:223] Registration of the crio container factory successfully
May 25 23:35:00.905387 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.905173    2578 factory.go:103] Registering Raw factory
May 25 23:35:00.905387 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.905189    2578 manager.go:1196] Started watching for new ooms in manager
May 25 23:35:00.906380 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.906364    2578 manager.go:319] Starting recovery of all containers
May 25 23:35:00.908056 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.908033    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vqx64"
May 25 23:35:00.917179 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.916983    2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 23:35:00.917830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.917815    2578 manager.go:324] Recovery completed
May 25 23:35:00.922843 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.922826    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 23:35:00.924761 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.924707    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
May 25 23:35:00.924843 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.924781    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
May 25 23:35:00.924843 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.924797    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
May 25 23:35:00.925291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.925276    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 23:35:00.925349 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.925292    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 23:35:00.925349 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.925310    2578 state_mem.go:36] "Initialized new in-memory state store"
May 25 23:35:00.927466 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.927454    2578 policy_none.go:49] "None policy: Start"
May 25 23:35:00.927506 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.927471    2578 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 23:35:00.927506 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.927482    2578 state_mem.go:35] "Initializing new in-memory state store"
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.968414    2578 manager.go:341] "Starting Device Plugin manager"
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:00.968451    2578 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.968463    2578 server.go:85] "Starting device plugin registration server"
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.968751    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.968763    2578 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.969489    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.969593    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:00.969604    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:00.970093    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 23:35:00.974596 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:00.970128    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.001468 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.001433    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 23:35:01.002668 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.002648    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 23:35:01.002764 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.002685    2578 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 23:35:01.002764 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.002708    2578 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 23:35:01.002764 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.002734    2578 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 23:35:01.002919 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.002788    2578 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 23:35:01.009967 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.009941    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 23:35:01.069838 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.069765    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 23:35:01.070846 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.070822    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
May 25 23:35:01.070961 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.070854    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
May 25 23:35:01.070961 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.070865    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
May 25 23:35:01.070961 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.070888    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-28.ec2.internal"
May 25 23:35:01.080706 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.080684    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-28.ec2.internal"
May 25 23:35:01.080825 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.080716    2578 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-28.ec2.internal\": node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.094276 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.094252    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.103678 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.103651    2578 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"]
May 25 23:35:01.103822 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.103756    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 23:35:01.105612 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.105597    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
May 25 23:35:01.105707 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.105631    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
May 25 23:35:01.105707 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.105645    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
May 25 23:35:01.106910 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.106894    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 23:35:01.107065 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.107050    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.107153 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.107082    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 23:35:01.107698 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.107675    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
May 25 23:35:01.107802 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.107704    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
May 25 23:35:01.107802 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.107714    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
May 25 23:35:01.107802 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.107675    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
May 25 23:35:01.107802 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.107768    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
May 25 23:35:01.107802 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.107781    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
May 25 23:35:01.108941 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.108927    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.108993 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.108953    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 23:35:01.109683 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.109669    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
May 25 23:35:01.109683 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.109690    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
May 25 23:35:01.109797 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.109700    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
May 25 23:35:01.134685 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.134653    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-28.ec2.internal\" not found" node="ip-10-0-130-28.ec2.internal"
May 25 23:35:01.139262 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.139244    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-28.ec2.internal\" not found" node="ip-10-0-130-28.ec2.internal"
May 25 23:35:01.194873 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.194840    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.295149 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.295109    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.301457 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.301429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b5caa621bb0dea4c45c2215675932558-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"b5caa621bb0dea4c45c2215675932558\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.301523 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.301465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5caa621bb0dea4c45c2215675932558-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"b5caa621bb0dea4c45c2215675932558\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.301523 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.301484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eebff1f1a969cd0c8699443c720bbf73-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"eebff1f1a969cd0c8699443c720bbf73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.395488 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.395416    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.401706 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.401690    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b5caa621bb0dea4c45c2215675932558-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"b5caa621bb0dea4c45c2215675932558\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.401787 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.401715    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5caa621bb0dea4c45c2215675932558-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"b5caa621bb0dea4c45c2215675932558\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.401787 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.401751    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eebff1f1a969cd0c8699443c720bbf73-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"eebff1f1a969cd0c8699443c720bbf73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.401850 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.401813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b5caa621bb0dea4c45c2215675932558-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"b5caa621bb0dea4c45c2215675932558\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.401881 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.401855    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5caa621bb0dea4c45c2215675932558-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"b5caa621bb0dea4c45c2215675932558\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.401916 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.401883    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eebff1f1a969cd0c8699443c720bbf73-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"eebff1f1a969cd0c8699443c720bbf73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.437935 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.437902    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.441656 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.441638    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
May 25 23:35:01.495920 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.495883    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.596417 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.596390    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.696946 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.696871    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.797392 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.797362    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.808851 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.808824    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 23:35:01.808985 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.808967    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 23:35:01.809023 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.809003    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 23:35:01.898011 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.897969    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:01.899812 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.899785    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 23:35:01.910247 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.910215    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 23:30:00 +0000 UTC" deadline="2027-11-05 15:28:51.08324008 +0000 UTC"
May 25 23:35:01.910247 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.910243    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12687h53m49.173000904s"
May 25 23:35:01.911628 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.911606    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 23:35:01.953320 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.953242    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q52m6"
May 25 23:35:01.963712 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:01.963682    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q52m6"
May 25 23:35:01.998440 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:01.998410    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:02.059630 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:02.059589    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5caa621bb0dea4c45c2215675932558.slice/crio-5ac67deae68c1881aec52fd503fb74f93cae03ba4c979295ea1bade228beffcd WatchSource:0}: Error finding container 5ac67deae68c1881aec52fd503fb74f93cae03ba4c979295ea1bade228beffcd: Status 404 returned error can't find the container with id 5ac67deae68c1881aec52fd503fb74f93cae03ba4c979295ea1bade228beffcd
May 25 23:35:02.059855 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:02.059833    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeebff1f1a969cd0c8699443c720bbf73.slice/crio-256d4ee164bb82bdc9cf36a20ec52564a5a799de48c10b0f96ff370d259836ea WatchSource:0}: Error finding container 256d4ee164bb82bdc9cf36a20ec52564a5a799de48c10b0f96ff370d259836ea: Status 404 returned error can't find the container with id 256d4ee164bb82bdc9cf36a20ec52564a5a799de48c10b0f96ff370d259836ea
May 25 23:35:02.063137 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.063121    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 23:35:02.099534 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:02.099494    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:02.200010 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:02.199965    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:02.300562 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:02.300480    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
May 25 23:35:02.353636 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.353612    2578 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 23:35:02.403012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.402983    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
May 25 23:35:02.423180 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.423148    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 23:35:02.425124 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.425107    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
May 25 23:35:02.442734 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.442703    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 23:35:02.454465 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.454439    2578 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 23:35:02.780560 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.780469    2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 23:35:02.881057 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.881005    2578 apiserver.go:52] "Watching apiserver"
May 25 23:35:02.888527 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.888497    2578 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 23:35:02.888989 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.888951    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-g24br","kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vr5m6","openshift-image-registry/node-ca-qhm9s","openshift-multus/multus-w7pqc","openshift-multus/network-metrics-daemon-lwvj8","openshift-network-operator/iptables-alerter-lklf2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal","openshift-multus/multus-additional-cni-plugins-h2c7q","openshift-network-diagnostics/network-check-target-zmrpl","openshift-ovn-kubernetes/ovnkube-node-ck9d2","kube-system/global-pull-secret-syncer-9s245"]
May 25 23:35:02.891764 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.891742    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:02.894106 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.893805    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.896214 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.896111    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:02.896355 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.896335    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 23:35:02.896482 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.896437    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 23:35:02.896545 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.896458    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 23:35:02.897554 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.897533    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r4qwk\""
May 25 23:35:02.897844 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.897825    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rqxl\""
May 25 23:35:02.901862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.898852    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 23:35:02.901862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.899184    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 23:35:02.901862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.901303    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 23:35:02.902077 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.901973    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 23:35:02.902488 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.902460    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 23:35:02.902980 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.902849    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pdgqq\""
May 25 23:35:02.904495 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.904473    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:02.904593 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:02.904550    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:02.906941 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.906883    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.907053 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.906984    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:02.908655 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.908633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-kubernetes\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.908866 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.908842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-run\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.909006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.908919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2tb\" (UniqueName: \"kubernetes.io/projected/0fd250b8-53ba-4220-847a-0eba50daa2d7-kube-api-access-2g2tb\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.909006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.908949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-lib-modules\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.909006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.908992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-tuned\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.909181 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-multus-cni-dir\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909181 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-cnibin\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909181 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-run-multus-certs\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909181 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5783a810-60b3-470b-a71e-98357f95b962-serviceca\") pod \"node-ca-qhm9s\" (UID: \"5783a810-60b3-470b-a71e-98357f95b962\") " pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:02.909377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-var-lib-cni-bin\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909220    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:02.909377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktb7\" (UniqueName: \"kubernetes.io/projected/ebb49344-a3fe-4a42-8075-62ead776d565-kube-api-access-7ktb7\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a27e83fe-fcf9-468d-ba60-ebc0746bc868-iptables-alerter-script\") pod \"iptables-alerter-lklf2\" (UID: \"a27e83fe-fcf9-468d-ba60-ebc0746bc868\") " pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:02.909377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a27e83fe-fcf9-468d-ba60-ebc0746bc868-host-slash\") pod \"iptables-alerter-lklf2\" (UID: \"a27e83fe-fcf9-468d-ba60-ebc0746bc868\") " pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:02.909377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhbb\" (UniqueName: \"kubernetes.io/projected/a27e83fe-fcf9-468d-ba60-ebc0746bc868-kube-api-access-kvhbb\") pod \"iptables-alerter-lklf2\" (UID: \"a27e83fe-fcf9-468d-ba60-ebc0746bc868\") " pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:02.909377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-sysctl-conf\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5783a810-60b3-470b-a71e-98357f95b962-host\") pod \"node-ca-qhm9s\" (UID: \"5783a810-60b3-470b-a71e-98357f95b962\") " pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-system-cni-dir\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-multus-socket-dir-parent\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-var-lib-cni-multus\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-os-release\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebb49344-a3fe-4a42-8075-62ead776d565-cni-binary-copy\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909583    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-sysctl-d\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-host\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-run-k8s-cni-cncf-io\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909673    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 23:35:02.909688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-run-netns\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909702    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-var-lib-kubelet\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-systemd\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-sys\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-sysconfig\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909823    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-multus-conf-dir\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4lq\" (UniqueName: \"kubernetes.io/projected/5783a810-60b3-470b-a71e-98357f95b962-kube-api-access-8x4lq\") pod \"node-ca-qhm9s\" (UID: \"5783a810-60b3-470b-a71e-98357f95b962\") " pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909919    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.909999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebb49344-a3fe-4a42-8075-62ead776d565-multus-daemon-config\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.910012    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jqsdf\""
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.910048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-modprobe-d\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.910086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-var-lib-kubelet\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.910128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fd250b8-53ba-4220-847a-0eba50daa2d7-tmp\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.910172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-hostroot\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.910177    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2xfn6\""
May 25 23:35:02.910248 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.910201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-etc-kubernetes\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:02.911061 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.910545    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 23:35:02.911803 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.911777    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 23:35:02.912394 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.911980    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 23:35:02.912986 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.912935    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 23:35:02.913180 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.913159    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x95qz\""
May 25 23:35:02.914520 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.914502    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:02.917247 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.917067    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 23:35:02.917338 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.917281    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:02.918147 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.918129    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b74tl\""
May 25 23:35:02.918343 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.918321    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 23:35:02.919701 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.919682    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 23:35:02.920595 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.919802    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:02.920595 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.919846    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:02.920595 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:02.919895    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:02.920595 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:02.919908    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:02.921467 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.921282    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 23:35:02.921467 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.921300    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 23:35:02.921467 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.921337    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 23:35:02.921657 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.921525    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 23:35:02.922100 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.921870    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 23:35:02.922100 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.921934    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qhwpb\""
May 25 23:35:02.964970 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.964932    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 23:30:01 +0000 UTC" deadline="2028-02-01 00:45:07.025555948 +0000 UTC"
May 25 23:35:02.964970 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:02.964968    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14785h10m4.060592505s"
May 25 23:35:03.001912 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.001871    2578 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 23:35:03.009144 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.009078    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" event={"ID":"eebff1f1a969cd0c8699443c720bbf73","Type":"ContainerStarted","Data":"256d4ee164bb82bdc9cf36a20ec52564a5a799de48c10b0f96ff370d259836ea"}
May 25 23:35:03.010220 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010186    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"b5caa621bb0dea4c45c2215675932558","Type":"ContainerStarted","Data":"5ac67deae68c1881aec52fd503fb74f93cae03ba4c979295ea1bade228beffcd"}
May 25 23:35:03.010418 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010396    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-var-lib-cni-bin\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010492 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010434    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktb7\" (UniqueName: \"kubernetes.io/projected/ebb49344-a3fe-4a42-8075-62ead776d565-kube-api-access-7ktb7\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010492 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-registration-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.010590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-run-netns\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.010590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010497    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-var-lib-cni-bin\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-cni-bin\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.010590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-dbus\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:03.010590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010563    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebb49344-a3fe-4a42-8075-62ead776d565-multus-daemon-config\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010582    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-system-cni-dir\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010597    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-var-lib-cni-multus\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010613    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebb49344-a3fe-4a42-8075-62ead776d565-cni-binary-copy\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010638    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-sysctl-d\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010662    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-host\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010664    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-system-cni-dir\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010687    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-run-k8s-cni-cncf-io\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010695    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-var-lib-cni-multus\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-socket-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-log-socket\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010797    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-host\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010809    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-sys\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010835    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-sysctl-d\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47dw\" (UniqueName: \"kubernetes.io/projected/d2a5c5ab-1582-42e5-9665-3a195365bb18-kube-api-access-x47dw\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010862    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-run-k8s-cni-cncf-io\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ab4c691-103b-4095-a962-d3c780650959-ovn-node-metrics-cert\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010898    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-sys\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.010979 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0ab4c691-103b-4095-a962-d3c780650959-ovnkube-script-lib\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010946    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-modprobe-d\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.010980    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fd250b8-53ba-4220-847a-0eba50daa2d7-tmp\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011015    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-hostroot\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-run-ovn\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011062    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-kubernetes\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011085    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2tb\" (UniqueName: \"kubernetes.io/projected/0fd250b8-53ba-4220-847a-0eba50daa2d7-kube-api-access-2g2tb\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011094    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-modprobe-d\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011139    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-hostroot\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-device-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011182    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-kubernetes\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25656d22-e01a-49b8-b274-0514816aae9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-kubelet\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011247    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebb49344-a3fe-4a42-8075-62ead776d565-multus-daemon-config\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-slash\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.011934 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011282    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebb49344-a3fe-4a42-8075-62ead776d565-cni-binary-copy\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011315    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-lib-modules\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011318    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011343    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-multus-cni-dir\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011368    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-cnibin\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011391    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-run-multus-certs\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-run-systemd\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0ab4c691-103b-4095-a962-d3c780650959-env-overrides\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-lib-modules\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011463    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5783a810-60b3-470b-a71e-98357f95b962-serviceca\") pod \"node-ca-qhm9s\" (UID: \"5783a810-60b3-470b-a71e-98357f95b962\") " pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011490    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-cnibin\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011523    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-run-multus-certs\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011551    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-multus-cni-dir\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25656d22-e01a-49b8-b274-0514816aae9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25656d22-e01a-49b8-b274-0514816aae9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-var-lib-openvswitch\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.012774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-system-cni-dir\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011779    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a27e83fe-fcf9-468d-ba60-ebc0746bc868-iptables-alerter-script\") pod \"iptables-alerter-lklf2\" (UID: \"a27e83fe-fcf9-468d-ba60-ebc0746bc868\") " pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011807    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a27e83fe-fcf9-468d-ba60-ebc0746bc868-host-slash\") pod \"iptables-alerter-lklf2\" (UID: \"a27e83fe-fcf9-468d-ba60-ebc0746bc868\") " pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011832    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhbb\" (UniqueName: \"kubernetes.io/projected/a27e83fe-fcf9-468d-ba60-ebc0746bc868-kube-api-access-kvhbb\") pod \"iptables-alerter-lklf2\" (UID: \"a27e83fe-fcf9-468d-ba60-ebc0746bc868\") " pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011858    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-sysctl-conf\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011884    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5783a810-60b3-470b-a71e-98357f95b962-host\") pod \"node-ca-qhm9s\" (UID: \"5783a810-60b3-470b-a71e-98357f95b962\") " pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011900    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5783a810-60b3-470b-a71e-98357f95b962-serviceca\") pod \"node-ca-qhm9s\" (UID: \"5783a810-60b3-470b-a71e-98357f95b962\") " pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011911    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-multus-socket-dir-parent\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011945    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-var-lib-kubelet\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-multus-socket-dir-parent\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011971    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-os-release\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.011997    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-run-netns\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/09660ea1-1664-415f-aa07-f0fef147096b-konnectivity-ca\") pod \"konnectivity-agent-g24br\" (UID: \"09660ea1-1664-415f-aa07-f0fef147096b\") " pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-etc-openvswitch\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-node-log\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:03.013513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012174    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-systemd\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a27e83fe-fcf9-468d-ba60-ebc0746bc868-host-slash\") pod \"iptables-alerter-lklf2\" (UID: \"a27e83fe-fcf9-468d-ba60-ebc0746bc868\") " pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012215    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-var-lib-kubelet\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012200    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-sysconfig\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012258    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-multus-conf-dir\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012263    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-sysconfig\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-multus-conf-dir\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012330    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-os-release\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012371    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-host-run-netns\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012419    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-sysctl-conf\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-os-release\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012453    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a27e83fe-fcf9-468d-ba60-ebc0746bc868-iptables-alerter-script\") pod \"iptables-alerter-lklf2\" (UID: \"a27e83fe-fcf9-468d-ba60-ebc0746bc868\") " pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012459    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-systemd\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qp4\" (UniqueName: \"kubernetes.io/projected/0ab4c691-103b-4095-a962-d3c780650959-kube-api-access-f4qp4\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012538    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4lq\" (UniqueName: \"kubernetes.io/projected/5783a810-60b3-470b-a71e-98357f95b962-kube-api-access-8x4lq\") pod \"node-ca-qhm9s\" (UID: \"5783a810-60b3-470b-a71e-98357f95b962\") " pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012547    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5783a810-60b3-470b-a71e-98357f95b962-host\") pod \"node-ca-qhm9s\" (UID: \"5783a810-60b3-470b-a71e-98357f95b962\") " pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f68j\" (UniqueName: \"kubernetes.io/projected/25656d22-e01a-49b8-b274-0514816aae9c-kube-api-access-8f68j\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.014415 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-run-openvswitch\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-run-ovn-kubernetes\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012771    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-var-lib-kubelet\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012812    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-etc-kubernetes\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012838    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebb49344-a3fe-4a42-8075-62ead776d565-etc-kubernetes\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmpc4\" (UniqueName: \"kubernetes.io/projected/accf65f4-6bda-4e39-ba44-c92d75f21e64-kube-api-access-kmpc4\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-etc-selinux\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-sys-fs\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/09660ea1-1664-415f-aa07-f0fef147096b-agent-certs\") pod \"konnectivity-agent-g24br\" (UID: \"09660ea1-1664-415f-aa07-f0fef147096b\") " pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.012999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-cni-netd\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.013030    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-var-lib-kubelet\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.013091    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-run\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.013129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-systemd-units\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.013177    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fd250b8-53ba-4220-847a-0eba50daa2d7-run\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.013190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0ab4c691-103b-4095-a962-d3c780650959-ovnkube-config\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.013236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-kubelet-config\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:03.016841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.013264    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-tuned\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.017478 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.013327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-cnibin\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.017478 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.015101    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fd250b8-53ba-4220-847a-0eba50daa2d7-tmp\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.017572 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.017559    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fd250b8-53ba-4220-847a-0eba50daa2d7-etc-tuned\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.024850 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.024793    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2tb\" (UniqueName: \"kubernetes.io/projected/0fd250b8-53ba-4220-847a-0eba50daa2d7-kube-api-access-2g2tb\") pod \"tuned-vr5m6\" (UID: \"0fd250b8-53ba-4220-847a-0eba50daa2d7\") " pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.024850 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.024826    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhbb\" (UniqueName: \"kubernetes.io/projected/a27e83fe-fcf9-468d-ba60-ebc0746bc868-kube-api-access-kvhbb\") pod \"iptables-alerter-lklf2\" (UID: \"a27e83fe-fcf9-468d-ba60-ebc0746bc868\") " pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:03.026446 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.026424    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktb7\" (UniqueName: \"kubernetes.io/projected/ebb49344-a3fe-4a42-8075-62ead776d565-kube-api-access-7ktb7\") pod \"multus-w7pqc\" (UID: \"ebb49344-a3fe-4a42-8075-62ead776d565\") " pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.026895 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.026870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4lq\" (UniqueName: \"kubernetes.io/projected/5783a810-60b3-470b-a71e-98357f95b962-kube-api-access-8x4lq\") pod \"node-ca-qhm9s\" (UID: \"5783a810-60b3-470b-a71e-98357f95b962\") " pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:03.114380 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114287    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-cnibin\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.114380 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114327    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-registration-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.114380 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114351    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-run-netns\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114396    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-cni-bin\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114420    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-dbus\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114436    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-registration-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114447    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-cnibin\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114475    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-socket-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114489    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-run-netns\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-cni-bin\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114501    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-log-socket\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114537    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-log-socket\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114542    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x47dw\" (UniqueName: \"kubernetes.io/projected/d2a5c5ab-1582-42e5-9665-3a195365bb18-kube-api-access-x47dw\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114575    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ab4c691-103b-4095-a962-d3c780650959-ovn-node-metrics-cert\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114601    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0ab4c691-103b-4095-a962-d3c780650959-ovnkube-script-lib\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-dbus\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114625    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-socket-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.114638 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114632    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-run-ovn\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114660    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114687    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-device-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114716    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25656d22-e01a-49b8-b274-0514816aae9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114765    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114799    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-kubelet\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114841    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-slash\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114846    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-device-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114871    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-run-systemd\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114892    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-run-ovn\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114895    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0ab4c691-103b-4095-a962-d3c780650959-env-overrides\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114935    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114963    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25656d22-e01a-49b8-b274-0514816aae9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.114991    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25656d22-e01a-49b8-b274-0514816aae9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115024    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-var-lib-openvswitch\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115047    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-system-cni-dir\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115083    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/09660ea1-1664-415f-aa07-f0fef147096b-konnectivity-ca\") pod \"konnectivity-agent-g24br\" (UID: \"09660ea1-1664-415f-aa07-f0fef147096b\") " pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:03.115309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115107    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-etc-openvswitch\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115131    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-node-log\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115142    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-kubelet\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-node-log\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115169    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115206    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115210    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115237    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-os-release\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115268    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qp4\" (UniqueName: \"kubernetes.io/projected/0ab4c691-103b-4095-a962-d3c780650959-kube-api-access-f4qp4\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-system-cni-dir\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0ab4c691-103b-4095-a962-d3c780650959-ovnkube-script-lib\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115333    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8f68j\" (UniqueName: \"kubernetes.io/projected/25656d22-e01a-49b8-b274-0514816aae9c-kube-api-access-8f68j\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115360    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-run-openvswitch\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115388    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-run-ovn-kubernetes\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115415    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115441    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kmpc4\" (UniqueName: \"kubernetes.io/projected/accf65f4-6bda-4e39-ba44-c92d75f21e64-kube-api-access-kmpc4\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:03.116135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115460    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25656d22-e01a-49b8-b274-0514816aae9c-os-release\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115469    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-etc-selinux\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115507    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-sys-fs\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.115510    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115535    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/09660ea1-1664-415f-aa07-f0fef147096b-agent-certs\") pod \"konnectivity-agent-g24br\" (UID: \"09660ea1-1664-415f-aa07-f0fef147096b\") " pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115562    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-cni-netd\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115578    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-etc-selinux\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.115605    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret podName:f0633bd5-0b65-4ce4-9fb5-963307feb3b6 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:03.615572705 +0000 UTC m=+3.170516987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret") pod "global-pull-secret-syncer-9s245" (UID: "f0633bd5-0b65-4ce4-9fb5-963307feb3b6") : object "kube-system"/"original-pull-secret" not registered
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115615    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-cni-netd\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115628    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-systemd-units\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115664    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0ab4c691-103b-4095-a962-d3c780650959-ovnkube-config\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115692    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-kubelet-config\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115768    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25656d22-e01a-49b8-b274-0514816aae9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115792    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-kubelet-config\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115826    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25656d22-e01a-49b8-b274-0514816aae9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115843    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0ab4c691-103b-4095-a962-d3c780650959-env-overrides\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.116830 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-run-ovn-kubernetes\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115895    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-var-lib-openvswitch\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115910    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-run-openvswitch\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115983    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-run-systemd\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115987    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-etc-openvswitch\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.115945    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-host-slash\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.116106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0ab4c691-103b-4095-a962-d3c780650959-systemd-units\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.116232    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/09660ea1-1664-415f-aa07-f0fef147096b-konnectivity-ca\") pod \"konnectivity-agent-g24br\" (UID: \"09660ea1-1664-415f-aa07-f0fef147096b\") " pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.116237    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2a5c5ab-1582-42e5-9665-3a195365bb18-sys-fs\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.116268    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0ab4c691-103b-4095-a962-d3c780650959-ovnkube-config\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.116290    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.116352    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs podName:accf65f4-6bda-4e39-ba44-c92d75f21e64 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:03.61633462 +0000 UTC m=+3.171278863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs") pod "network-metrics-daemon-lwvj8" (UID: "accf65f4-6bda-4e39-ba44-c92d75f21e64") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.116489    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25656d22-e01a-49b8-b274-0514816aae9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.117514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.117453    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ab4c691-103b-4095-a962-d3c780650959-ovn-node-metrics-cert\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.118225 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.118205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/09660ea1-1664-415f-aa07-f0fef147096b-agent-certs\") pod \"konnectivity-agent-g24br\" (UID: \"09660ea1-1664-415f-aa07-f0fef147096b\") " pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:03.133996 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.133945    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47dw\" (UniqueName: \"kubernetes.io/projected/d2a5c5ab-1582-42e5-9665-3a195365bb18-kube-api-access-x47dw\") pod \"aws-ebs-csi-driver-node-xt6ft\" (UID: \"d2a5c5ab-1582-42e5-9665-3a195365bb18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.145143 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.145115    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 23:35:03.145278 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.145148    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 23:35:03.145278 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.145165    2578 projected.go:194] Error preparing data for projected volume kube-api-access-lhlvg for pod openshift-network-diagnostics/network-check-target-zmrpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:03.145278 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.145227    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg podName:d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d nodeName:}" failed. No retries permitted until 2026-05-25 23:35:03.645212447 +0000 UTC m=+3.200156690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lhlvg" (UniqueName: "kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg") pod "network-check-target-zmrpl" (UID: "d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:03.147574 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.147550    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f68j\" (UniqueName: \"kubernetes.io/projected/25656d22-e01a-49b8-b274-0514816aae9c-kube-api-access-8f68j\") pod \"multus-additional-cni-plugins-h2c7q\" (UID: \"25656d22-e01a-49b8-b274-0514816aae9c\") " pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.147687 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.147599    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qp4\" (UniqueName: \"kubernetes.io/projected/0ab4c691-103b-4095-a962-d3c780650959-kube-api-access-f4qp4\") pod \"ovnkube-node-ck9d2\" (UID: \"0ab4c691-103b-4095-a962-d3c780650959\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.149578 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.149557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmpc4\" (UniqueName: \"kubernetes.io/projected/accf65f4-6bda-4e39-ba44-c92d75f21e64-kube-api-access-kmpc4\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:03.208527 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.208495    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lklf2"
May 25 23:35:03.216008 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.215985    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vr5m6"
May 25 23:35:03.226371 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.226348    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qhm9s"
May 25 23:35:03.231842 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.231821    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7pqc"
May 25 23:35:03.239424 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.239401    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:03.248132 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.248112    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft"
May 25 23:35:03.254736 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.254701    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2c7q"
May 25 23:35:03.260388 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.260368    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:03.317736 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.317696    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 23:35:03.620584 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.620552    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:03.620769 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.620594    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:03.620769 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.620744    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:03.620769 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.620750    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 23:35:03.620913 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.620799    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs podName:accf65f4-6bda-4e39-ba44-c92d75f21e64 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:04.620786572 +0000 UTC m=+4.175730811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs") pod "network-metrics-daemon-lwvj8" (UID: "accf65f4-6bda-4e39-ba44-c92d75f21e64") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:03.620913 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.620824    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret podName:f0633bd5-0b65-4ce4-9fb5-963307feb3b6 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:04.620814529 +0000 UTC m=+4.175758771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret") pod "global-pull-secret-syncer-9s245" (UID: "f0633bd5-0b65-4ce4-9fb5-963307feb3b6") : object "kube-system"/"original-pull-secret" not registered
May 25 23:35:03.721550 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.721510    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:03.721754 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.721694    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 23:35:03.721754 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.721717    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 23:35:03.721754 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.721744    2578 projected.go:194] Error preparing data for projected volume kube-api-access-lhlvg for pod openshift-network-diagnostics/network-check-target-zmrpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:03.721911 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:03.721817    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg podName:d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d nodeName:}" failed. No retries permitted until 2026-05-25 23:35:04.721796382 +0000 UTC m=+4.276740643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhlvg" (UniqueName: "kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg") pod "network-check-target-zmrpl" (UID: "d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:03.830099 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:03.830068    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09660ea1_1664_415f_aa07_f0fef147096b.slice/crio-b31c99f443945be12f5e2b3b7e698d5fb3834eecb6e13e56400013dd02935ae2 WatchSource:0}: Error finding container b31c99f443945be12f5e2b3b7e698d5fb3834eecb6e13e56400013dd02935ae2: Status 404 returned error can't find the container with id b31c99f443945be12f5e2b3b7e698d5fb3834eecb6e13e56400013dd02935ae2
May 25 23:35:03.831532 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:03.831403    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd250b8_53ba_4220_847a_0eba50daa2d7.slice/crio-48150daf7a10f58f03140b4dfac4024d7318db18aefaac41f52267168d193e85 WatchSource:0}: Error finding container 48150daf7a10f58f03140b4dfac4024d7318db18aefaac41f52267168d193e85: Status 404 returned error can't find the container with id 48150daf7a10f58f03140b4dfac4024d7318db18aefaac41f52267168d193e85
May 25 23:35:03.833393 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:03.833279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb49344_a3fe_4a42_8075_62ead776d565.slice/crio-7d6092234b3b721cd59e40cf0462945f4ec89ef34ac8cabccd15d5201e156354 WatchSource:0}: Error finding container 7d6092234b3b721cd59e40cf0462945f4ec89ef34ac8cabccd15d5201e156354: Status 404 returned error can't find the container with id 7d6092234b3b721cd59e40cf0462945f4ec89ef34ac8cabccd15d5201e156354
May 25 23:35:03.840265 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:03.836513    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab4c691_103b_4095_a962_d3c780650959.slice/crio-593213184f1cb9b8268c6c0580e5c35c79e59c0d355a47a6197909ae1eb007b7 WatchSource:0}: Error finding container 593213184f1cb9b8268c6c0580e5c35c79e59c0d355a47a6197909ae1eb007b7: Status 404 returned error can't find the container with id 593213184f1cb9b8268c6c0580e5c35c79e59c0d355a47a6197909ae1eb007b7
May 25 23:35:03.842127 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:03.842100    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27e83fe_fcf9_468d_ba60_ebc0746bc868.slice/crio-af8cd45e0ad7bc1541441951400ac2f54604dba25cc3770c07dcffd55ff02341 WatchSource:0}: Error finding container af8cd45e0ad7bc1541441951400ac2f54604dba25cc3770c07dcffd55ff02341: Status 404 returned error can't find the container with id af8cd45e0ad7bc1541441951400ac2f54604dba25cc3770c07dcffd55ff02341
May 25 23:35:03.842882 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:03.842855    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a5c5ab_1582_42e5_9665_3a195365bb18.slice/crio-17084820b99996d489a9096621435aa64957a11ed5cc7f89f6587453c950e58d WatchSource:0}: Error finding container 17084820b99996d489a9096621435aa64957a11ed5cc7f89f6587453c950e58d: Status 404 returned error can't find the container with id 17084820b99996d489a9096621435aa64957a11ed5cc7f89f6587453c950e58d
May 25 23:35:03.843568 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:03.843548    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5783a810_60b3_470b_a71e_98357f95b962.slice/crio-9dcc4b53ff8b012432e7d95edae9cd2daaf00f201bf97cee6b621d44fb91e9ae WatchSource:0}: Error finding container 9dcc4b53ff8b012432e7d95edae9cd2daaf00f201bf97cee6b621d44fb91e9ae: Status 404 returned error can't find the container with id 9dcc4b53ff8b012432e7d95edae9cd2daaf00f201bf97cee6b621d44fb91e9ae
May 25 23:35:03.844319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:03.844299    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25656d22_e01a_49b8_b274_0514816aae9c.slice/crio-08b4ef77b953385cef5220797f2a8cc8d4d547a47046538cbd7097bdf8b1b0c2 WatchSource:0}: Error finding container 08b4ef77b953385cef5220797f2a8cc8d4d547a47046538cbd7097bdf8b1b0c2: Status 404 returned error can't find the container with id 08b4ef77b953385cef5220797f2a8cc8d4d547a47046538cbd7097bdf8b1b0c2
May 25 23:35:03.966207 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.966012    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 23:30:01 +0000 UTC" deadline="2028-02-23 22:06:05.103277932 +0000 UTC"
May 25 23:35:03.966207 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:03.966202    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15334h31m1.137079678s"
May 25 23:35:04.013425 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.013377    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lklf2" event={"ID":"a27e83fe-fcf9-468d-ba60-ebc0746bc868","Type":"ContainerStarted","Data":"af8cd45e0ad7bc1541441951400ac2f54604dba25cc3770c07dcffd55ff02341"}
May 25 23:35:04.015764 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.015715    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qhm9s" event={"ID":"5783a810-60b3-470b-a71e-98357f95b962","Type":"ContainerStarted","Data":"9dcc4b53ff8b012432e7d95edae9cd2daaf00f201bf97cee6b621d44fb91e9ae"}
May 25 23:35:04.019828 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.019798    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft" event={"ID":"d2a5c5ab-1582-42e5-9665-3a195365bb18","Type":"ContainerStarted","Data":"17084820b99996d489a9096621435aa64957a11ed5cc7f89f6587453c950e58d"}
May 25 23:35:04.020880 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.020855    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vr5m6" event={"ID":"0fd250b8-53ba-4220-847a-0eba50daa2d7","Type":"ContainerStarted","Data":"48150daf7a10f58f03140b4dfac4024d7318db18aefaac41f52267168d193e85"}
May 25 23:35:04.021816 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.021794    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" event={"ID":"25656d22-e01a-49b8-b274-0514816aae9c","Type":"ContainerStarted","Data":"08b4ef77b953385cef5220797f2a8cc8d4d547a47046538cbd7097bdf8b1b0c2"}
May 25 23:35:04.022753 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.022715    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerStarted","Data":"593213184f1cb9b8268c6c0580e5c35c79e59c0d355a47a6197909ae1eb007b7"}
May 25 23:35:04.024409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.024384    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7pqc" event={"ID":"ebb49344-a3fe-4a42-8075-62ead776d565","Type":"ContainerStarted","Data":"7d6092234b3b721cd59e40cf0462945f4ec89ef34ac8cabccd15d5201e156354"}
May 25 23:35:04.026662 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.026607    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g24br" event={"ID":"09660ea1-1664-415f-aa07-f0fef147096b","Type":"ContainerStarted","Data":"b31c99f443945be12f5e2b3b7e698d5fb3834eecb6e13e56400013dd02935ae2"}
May 25 23:35:04.028751 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.028712    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" event={"ID":"eebff1f1a969cd0c8699443c720bbf73","Type":"ContainerStarted","Data":"369e2e6184e1fc666946abd64fded72e2436bf2ef5f25eeff6bae5fa4f6e50b7"}
May 25 23:35:04.045372 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.045311    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" podStartSLOduration=2.04529167 podStartE2EDuration="2.04529167s" podCreationTimestamp="2026-05-25 23:35:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:35:04.044846337 +0000 UTC m=+3.599790601" watchObservedRunningTime="2026-05-25 23:35:04.04529167 +0000 UTC m=+3.600235934"
May 25 23:35:04.628380 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.628342    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:04.628577 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.628399    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:04.628577 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:04.628540    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:04.628684 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:04.628603    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs podName:accf65f4-6bda-4e39-ba44-c92d75f21e64 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:06.628585251 +0000 UTC m=+6.183529493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs") pod "network-metrics-daemon-lwvj8" (UID: "accf65f4-6bda-4e39-ba44-c92d75f21e64") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:04.629059 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:04.629038    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 23:35:04.629163 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:04.629099    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret podName:f0633bd5-0b65-4ce4-9fb5-963307feb3b6 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:06.629080783 +0000 UTC m=+6.184025025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret") pod "global-pull-secret-syncer-9s245" (UID: "f0633bd5-0b65-4ce4-9fb5-963307feb3b6") : object "kube-system"/"original-pull-secret" not registered
May 25 23:35:04.729106 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:04.729066    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:04.729276 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:04.729250    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 23:35:04.729276 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:04.729269    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 23:35:04.729389 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:04.729281    2578 projected.go:194] Error preparing data for projected volume kube-api-access-lhlvg for pod openshift-network-diagnostics/network-check-target-zmrpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:04.729389 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:04.729342    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg podName:d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d nodeName:}" failed. No retries permitted until 2026-05-25 23:35:06.729323382 +0000 UTC m=+6.284267638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhlvg" (UniqueName: "kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg") pod "network-check-target-zmrpl" (UID: "d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:05.003933 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:05.003310    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:05.003933 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:05.003458    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:05.003933 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:05.003758    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:05.003933 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:05.003869    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:05.004798 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:05.004641    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:05.004798 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:05.004757    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:05.058887 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:05.058847    2578 generic.go:358] "Generic (PLEG): container finished" podID="b5caa621bb0dea4c45c2215675932558" containerID="4b217456b558d89b8c77efb8987c4acb8165f86a0d06da5bac68ae13633a4177" exitCode=0
May 25 23:35:05.060130 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:05.059774    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"b5caa621bb0dea4c45c2215675932558","Type":"ContainerDied","Data":"4b217456b558d89b8c77efb8987c4acb8165f86a0d06da5bac68ae13633a4177"}
May 25 23:35:06.072892 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:06.072829    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"b5caa621bb0dea4c45c2215675932558","Type":"ContainerStarted","Data":"cedb2517eb88777174d05cae0df6b5828753841e47bfbe1f5f6af188c75224c1"}
May 25 23:35:06.646712 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:06.646505    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:06.646712 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:06.646561    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:06.646960 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:06.646717    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:06.646960 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:06.646804    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs podName:accf65f4-6bda-4e39-ba44-c92d75f21e64 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:10.646782767 +0000 UTC m=+10.201727011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs") pod "network-metrics-daemon-lwvj8" (UID: "accf65f4-6bda-4e39-ba44-c92d75f21e64") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:06.647146 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:06.647124    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 23:35:06.647222 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:06.647192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret podName:f0633bd5-0b65-4ce4-9fb5-963307feb3b6 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:10.647175117 +0000 UTC m=+10.202119373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret") pod "global-pull-secret-syncer-9s245" (UID: "f0633bd5-0b65-4ce4-9fb5-963307feb3b6") : object "kube-system"/"original-pull-secret" not registered
May 25 23:35:06.747547 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:06.747506    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:06.747745 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:06.747703    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 23:35:06.747745 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:06.747741    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 23:35:06.747941 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:06.747754    2578 projected.go:194] Error preparing data for projected volume kube-api-access-lhlvg for pod openshift-network-diagnostics/network-check-target-zmrpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:06.747941 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:06.747813    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg podName:d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d nodeName:}" failed. No retries permitted until 2026-05-25 23:35:10.747796199 +0000 UTC m=+10.302740439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhlvg" (UniqueName: "kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg") pod "network-check-target-zmrpl" (UID: "d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:07.003693 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:07.003608    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:07.003902 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:07.003772    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:07.003902 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:07.003850    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:07.003902 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:07.003765    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:07.003902 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:07.003608    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:07.004117 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:07.003937    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:09.007226 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:09.007178    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:09.007226 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:09.007206    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:09.007778 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:09.007178    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:09.007778 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:09.007306    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:09.007778 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:09.007397    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:09.007778 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:09.007485    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:10.680881 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:10.680779    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:10.680881 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:10.680839    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:10.681422 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:10.680975    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 23:35:10.681422 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:10.680990    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:10.681422 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:10.681061    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs podName:accf65f4-6bda-4e39-ba44-c92d75f21e64 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:18.681041276 +0000 UTC m=+18.235985532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs") pod "network-metrics-daemon-lwvj8" (UID: "accf65f4-6bda-4e39-ba44-c92d75f21e64") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:10.681422 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:10.681087    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret podName:f0633bd5-0b65-4ce4-9fb5-963307feb3b6 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:18.681072679 +0000 UTC m=+18.236016922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret") pod "global-pull-secret-syncer-9s245" (UID: "f0633bd5-0b65-4ce4-9fb5-963307feb3b6") : object "kube-system"/"original-pull-secret" not registered
May 25 23:35:10.781674 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:10.781470    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:10.781889 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:10.781693    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 23:35:10.781889 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:10.781713    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 23:35:10.781889 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:10.781742    2578 projected.go:194] Error preparing data for projected volume kube-api-access-lhlvg for pod openshift-network-diagnostics/network-check-target-zmrpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:10.781889 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:10.781810    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg podName:d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d nodeName:}" failed. No retries permitted until 2026-05-25 23:35:18.781791042 +0000 UTC m=+18.336735284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhlvg" (UniqueName: "kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg") pod "network-check-target-zmrpl" (UID: "d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:11.004804 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:11.004002    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:11.004804 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:11.004115    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:11.004804 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:11.004490    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:11.004804 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:11.004596    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:11.004804 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:11.004642    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:11.004804 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:11.004709    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:13.007243 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:13.007208    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:13.007708 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:13.007211    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:13.007708 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:13.007327    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:13.007708 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:13.007211    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:13.007708 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:13.007481    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:13.007708 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:13.007549    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:14.994838 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:14.994786    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" podStartSLOduration=12.994768626 podStartE2EDuration="12.994768626s" podCreationTimestamp="2026-05-25 23:35:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:35:06.089752432 +0000 UTC m=+5.644696694" watchObservedRunningTime="2026-05-25 23:35:14.994768626 +0000 UTC m=+14.549712874"
May 25 23:35:14.995318 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:14.995159    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-fnfhv"]
May 25 23:35:14.999329 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:14.999297    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.001787 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.001761    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 23:35:15.001787 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.001771    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5hv5\""
May 25 23:35:15.002599 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.002578    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 23:35:15.005614 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.005594    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:15.005614 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.005610    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:15.005765 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.005599    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:15.005765 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:15.005703    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:15.005933 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:15.005818    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:15.005933 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:15.005901    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:15.116024 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.115994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c0947b-069f-4202-bfff-2b66c658a943-tmp-dir\") pod \"node-resolver-fnfhv\" (UID: \"a5c0947b-069f-4202-bfff-2b66c658a943\") " pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.116188 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.116029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l54d\" (UniqueName: \"kubernetes.io/projected/a5c0947b-069f-4202-bfff-2b66c658a943-kube-api-access-5l54d\") pod \"node-resolver-fnfhv\" (UID: \"a5c0947b-069f-4202-bfff-2b66c658a943\") " pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.116188 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.116128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5c0947b-069f-4202-bfff-2b66c658a943-hosts-file\") pod \"node-resolver-fnfhv\" (UID: \"a5c0947b-069f-4202-bfff-2b66c658a943\") " pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.217406 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.217369    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5c0947b-069f-4202-bfff-2b66c658a943-hosts-file\") pod \"node-resolver-fnfhv\" (UID: \"a5c0947b-069f-4202-bfff-2b66c658a943\") " pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.217579 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.217459    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c0947b-069f-4202-bfff-2b66c658a943-tmp-dir\") pod \"node-resolver-fnfhv\" (UID: \"a5c0947b-069f-4202-bfff-2b66c658a943\") " pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.217579 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.217488    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5l54d\" (UniqueName: \"kubernetes.io/projected/a5c0947b-069f-4202-bfff-2b66c658a943-kube-api-access-5l54d\") pod \"node-resolver-fnfhv\" (UID: \"a5c0947b-069f-4202-bfff-2b66c658a943\") " pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.217579 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.217515    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5c0947b-069f-4202-bfff-2b66c658a943-hosts-file\") pod \"node-resolver-fnfhv\" (UID: \"a5c0947b-069f-4202-bfff-2b66c658a943\") " pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.217842 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.217821    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c0947b-069f-4202-bfff-2b66c658a943-tmp-dir\") pod \"node-resolver-fnfhv\" (UID: \"a5c0947b-069f-4202-bfff-2b66c658a943\") " pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.235516 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.235486    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l54d\" (UniqueName: \"kubernetes.io/projected/a5c0947b-069f-4202-bfff-2b66c658a943-kube-api-access-5l54d\") pod \"node-resolver-fnfhv\" (UID: \"a5c0947b-069f-4202-bfff-2b66c658a943\") " pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:15.309535 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:15.309496    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fnfhv"
May 25 23:35:17.006645 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:17.006617    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:17.007101 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:17.006624    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:17.007101 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:17.006712    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:17.007101 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:17.006624    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:17.007101 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:17.006840    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:17.007101 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:17.006945    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:18.743090 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:18.743044    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:18.743648 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:18.743113    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:18.743648 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:18.743228    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 23:35:18.743648 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:18.743264    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:18.743648 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:18.743308    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret podName:f0633bd5-0b65-4ce4-9fb5-963307feb3b6 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:34.743286679 +0000 UTC m=+34.298230938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret") pod "global-pull-secret-syncer-9s245" (UID: "f0633bd5-0b65-4ce4-9fb5-963307feb3b6") : object "kube-system"/"original-pull-secret" not registered
May 25 23:35:18.743648 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:18.743332    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs podName:accf65f4-6bda-4e39-ba44-c92d75f21e64 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:34.743320753 +0000 UTC m=+34.298265013 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs") pod "network-metrics-daemon-lwvj8" (UID: "accf65f4-6bda-4e39-ba44-c92d75f21e64") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:18.844315 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:18.844269    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:18.844515 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:18.844458    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 23:35:18.844515 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:18.844482    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 23:35:18.844515 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:18.844495    2578 projected.go:194] Error preparing data for projected volume kube-api-access-lhlvg for pod openshift-network-diagnostics/network-check-target-zmrpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:18.844647 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:18.844562    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg podName:d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d nodeName:}" failed. No retries permitted until 2026-05-25 23:35:34.844539439 +0000 UTC m=+34.399483685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhlvg" (UniqueName: "kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg") pod "network-check-target-zmrpl" (UID: "d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:19.007006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:19.006927    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:19.007176 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:19.006927    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:19.007176 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:19.007041    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:19.007176 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:19.007122    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:19.007176 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:19.006927    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:19.007359 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:19.007221    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:21.004923 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:21.004881    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:21.005374 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:21.004977    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:21.005374 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:21.005025    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:21.005374 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:21.005029    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:21.005374 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:21.005110    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:21.005374 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:21.005182    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:21.687319 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:21.687289    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c0947b_069f_4202_bfff_2b66c658a943.slice/crio-15f01f76ebf8e68c263a259f68ee900a13e3923e86c29e8cace96b80e81006a3 WatchSource:0}: Error finding container 15f01f76ebf8e68c263a259f68ee900a13e3923e86c29e8cace96b80e81006a3: Status 404 returned error can't find the container with id 15f01f76ebf8e68c263a259f68ee900a13e3923e86c29e8cace96b80e81006a3
May 25 23:35:22.100996 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.100670    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" event={"ID":"25656d22-e01a-49b8-b274-0514816aae9c","Type":"ContainerStarted","Data":"2e09911799aca7ead197c620380e0759cb99e602e2984ffc394ab7a996aa373f"}
May 25 23:35:22.102656 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.102622    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerStarted","Data":"be3b5166dd15c4e6623bb9dd56ae0e9ed76b647b2f35c8d0436ec345d64683ae"}
May 25 23:35:22.104179 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.104154    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7pqc" event={"ID":"ebb49344-a3fe-4a42-8075-62ead776d565","Type":"ContainerStarted","Data":"f71210085ec5efd6a975218de58443482f3ef4c29afa24149ed353f7686af9c2"}
May 25 23:35:22.105609 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.105515    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g24br" event={"ID":"09660ea1-1664-415f-aa07-f0fef147096b","Type":"ContainerStarted","Data":"be768580e232e42ca651423a5543289ed22cdc2e6a97822a28de9cf2d57fa00f"}
May 25 23:35:22.107259 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.107243    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qhm9s" event={"ID":"5783a810-60b3-470b-a71e-98357f95b962","Type":"ContainerStarted","Data":"1c9832a965e2d92d7ae318b1e6e65c4dfd9f2efe2cc649d783e1d31040774244"}
May 25 23:35:22.108732 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.108694    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft" event={"ID":"d2a5c5ab-1582-42e5-9665-3a195365bb18","Type":"ContainerStarted","Data":"8af4aa5f2fdb14ec9e1791e46a2d2fc5d1be4e61ce4aa4aac7351faedfa3fad8"}
May 25 23:35:22.110196 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.110172    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vr5m6" event={"ID":"0fd250b8-53ba-4220-847a-0eba50daa2d7","Type":"ContainerStarted","Data":"aa9ab1d266edaf66dbbe75bbd883d9b7635963a6544b8267ed08c86bd42e6a93"}
May 25 23:35:22.111342 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.111323    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fnfhv" event={"ID":"a5c0947b-069f-4202-bfff-2b66c658a943","Type":"ContainerStarted","Data":"52d67fe11797d336ef2aabccbca85a2e72fdc1d6853430acbf5efd1693a2e498"}
May 25 23:35:22.111420 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.111347    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fnfhv" event={"ID":"a5c0947b-069f-4202-bfff-2b66c658a943","Type":"ContainerStarted","Data":"15f01f76ebf8e68c263a259f68ee900a13e3923e86c29e8cace96b80e81006a3"}
May 25 23:35:22.140829 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.140788    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-fnfhv" podStartSLOduration=8.140771241 podStartE2EDuration="8.140771241s" podCreationTimestamp="2026-05-25 23:35:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:35:22.140464746 +0000 UTC m=+21.695409007" watchObservedRunningTime="2026-05-25 23:35:22.140771241 +0000 UTC m=+21.695715502"
May 25 23:35:22.165998 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.164491    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-w7pqc" podStartSLOduration=3.300419343 podStartE2EDuration="21.164432775s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:35:03.835131987 +0000 UTC m=+3.390076230" lastFinishedPulling="2026-05-25 23:35:21.699145423 +0000 UTC m=+21.254089662" observedRunningTime="2026-05-25 23:35:22.163161016 +0000 UTC m=+21.718105278" watchObservedRunningTime="2026-05-25 23:35:22.164432775 +0000 UTC m=+21.719377038"
May 25 23:35:22.187814 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.187766    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g24br" podStartSLOduration=8.130135685 podStartE2EDuration="21.187751386s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:35:03.831763062 +0000 UTC m=+3.386707300" lastFinishedPulling="2026-05-25 23:35:16.889378753 +0000 UTC m=+16.444323001" observedRunningTime="2026-05-25 23:35:22.187467252 +0000 UTC m=+21.742411520" watchObservedRunningTime="2026-05-25 23:35:22.187751386 +0000 UTC m=+21.742695643"
May 25 23:35:22.213163 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.213068    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vr5m6" podStartSLOduration=3.364102822 podStartE2EDuration="21.213052464s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:35:03.833981097 +0000 UTC m=+3.388925350" lastFinishedPulling="2026-05-25 23:35:21.682930736 +0000 UTC m=+21.237874992" observedRunningTime="2026-05-25 23:35:22.212510508 +0000 UTC m=+21.767454769" watchObservedRunningTime="2026-05-25 23:35:22.213052464 +0000 UTC m=+21.767996725"
May 25 23:35:22.237645 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.237592    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qhm9s" podStartSLOduration=3.402589175 podStartE2EDuration="21.23757276s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:35:03.845627964 +0000 UTC m=+3.400572217" lastFinishedPulling="2026-05-25 23:35:21.680611554 +0000 UTC m=+21.235555802" observedRunningTime="2026-05-25 23:35:22.235629636 +0000 UTC m=+21.790573877" watchObservedRunningTime="2026-05-25 23:35:22.23757276 +0000 UTC m=+21.792517025"
May 25 23:35:22.400617 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.400393    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:22.401068 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:22.401049    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:23.003278 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.003196    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:23.003461 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.003196    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:23.003461 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:23.003305    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:23.003461 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.003196    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:23.003461 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:23.003384    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:23.003592 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:23.003459    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:23.114898 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.114860    2578 generic.go:358] "Generic (PLEG): container finished" podID="25656d22-e01a-49b8-b274-0514816aae9c" containerID="2e09911799aca7ead197c620380e0759cb99e602e2984ffc394ab7a996aa373f" exitCode=0
May 25 23:35:23.115561 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.114947    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" event={"ID":"25656d22-e01a-49b8-b274-0514816aae9c","Type":"ContainerDied","Data":"2e09911799aca7ead197c620380e0759cb99e602e2984ffc394ab7a996aa373f"}
May 25 23:35:23.117500 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.117480    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:35:23.117801 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.117780    2578 generic.go:358] "Generic (PLEG): container finished" podID="0ab4c691-103b-4095-a962-d3c780650959" containerID="3894eb03a857e761607bbf8cd65b217fa845ff24245743c407bc5c8bbd260b5e" exitCode=1
May 25 23:35:23.117878 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.117859    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerDied","Data":"3894eb03a857e761607bbf8cd65b217fa845ff24245743c407bc5c8bbd260b5e"}
May 25 23:35:23.117919 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.117903    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerStarted","Data":"f1bade4e96fc6f6997a8009d90c6aaddb13c8987c78b4034c34f40739e61bc53"}
May 25 23:35:23.117953 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.117920    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerStarted","Data":"51a6c9e63e9b543cae6f03bcd02bb3d76051231ec62a9718680f08ab03a220ef"}
May 25 23:35:23.117953 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.117936    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerStarted","Data":"2e6211e69749aa2fca1957171df1fbcbe4d094463c5fb492ac9f32d32e78d74a"}
May 25 23:35:23.117953 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.117946    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerStarted","Data":"ab87b5ff468b62ac55ff8bc0166d1472f1cd9c6ca1da6edadddfc3413e902357"}
May 25 23:35:23.118500 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.118481    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:23.118703 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.118689    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g24br"
May 25 23:35:23.299365 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.299335    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 23:35:23.982606 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.982411    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T23:35:23.299360102Z","UUID":"5a4bdffa-b845-4a8e-a93e-8b2c6899f3cf","Handler":null,"Name":"","Endpoint":""}
May 25 23:35:23.984641 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.984610    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 23:35:23.984641 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:23.984645    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 23:35:24.122225 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:24.122170    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lklf2" event={"ID":"a27e83fe-fcf9-468d-ba60-ebc0746bc868","Type":"ContainerStarted","Data":"295da2a5615f9bdaff5977e1f5cbff4393c4bde92583b2cdc32475e83bd10a97"}
May 25 23:35:24.124770 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:24.124736    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft" event={"ID":"d2a5c5ab-1582-42e5-9665-3a195365bb18","Type":"ContainerStarted","Data":"2c5b2020c00ec8771ec781b6a67194a70e10775c1b0a0f8f8c0425f5089a3eac"}
May 25 23:35:24.140841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:24.140785    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lklf2" podStartSLOduration=5.3017346530000005 podStartE2EDuration="23.140765627s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:35:03.843909841 +0000 UTC m=+3.398854084" lastFinishedPulling="2026-05-25 23:35:21.682940812 +0000 UTC m=+21.237885058" observedRunningTime="2026-05-25 23:35:24.140110462 +0000 UTC m=+23.695054725" watchObservedRunningTime="2026-05-25 23:35:24.140765627 +0000 UTC m=+23.695709890"
May 25 23:35:25.003898 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:25.003613    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:25.004088 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:25.003617    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:25.004088 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:25.003624    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:25.004576 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:25.004544    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:25.004743 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:25.004603    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:25.004743 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:25.004702    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:25.129255 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:25.129211    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft" event={"ID":"d2a5c5ab-1582-42e5-9665-3a195365bb18","Type":"ContainerStarted","Data":"079d2b96fee51f2a458f07962773a1d97ec0df1f35f076a4a9b150a0d332c93b"}
May 25 23:35:25.132399 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:25.132374    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:35:25.133258 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:25.133216    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerStarted","Data":"826b942da49ab73debb157c482195ec631037ee6f986b864c73995d48466b3b7"}
May 25 23:35:25.149667 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:25.149606    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt6ft" podStartSLOduration=3.352314903 podStartE2EDuration="24.149591918s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:35:03.845303274 +0000 UTC m=+3.400247516" lastFinishedPulling="2026-05-25 23:35:24.642580283 +0000 UTC m=+24.197524531" observedRunningTime="2026-05-25 23:35:25.14925968 +0000 UTC m=+24.704203962" watchObservedRunningTime="2026-05-25 23:35:25.149591918 +0000 UTC m=+24.704536178"
May 25 23:35:27.003192 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:27.003160    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:27.003918 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:27.003161    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:27.003918 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:27.003282    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:27.003918 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:27.003161    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:27.003918 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:27.003348    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:27.003918 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:27.003473    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:28.140305 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:28.140085    2578 generic.go:358] "Generic (PLEG): container finished" podID="25656d22-e01a-49b8-b274-0514816aae9c" containerID="dfb2f33652600f1b8d15fc132bae29f343199b464e56d7adf9651f53d886d322" exitCode=0
May 25 23:35:28.141044 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:28.140170    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" event={"ID":"25656d22-e01a-49b8-b274-0514816aae9c","Type":"ContainerDied","Data":"dfb2f33652600f1b8d15fc132bae29f343199b464e56d7adf9651f53d886d322"}
May 25 23:35:28.143410 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:28.143390    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:35:28.143711 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:28.143691    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerStarted","Data":"e65dd0785b0dff63d7a7b4fc76d0b0dc8d13391a1970d69e62c6b74d217003ba"}
May 25 23:35:28.144060 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:28.144042    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:28.144158 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:28.144068    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:28.144199 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:28.144189    2578 scope.go:117] "RemoveContainer" containerID="3894eb03a857e761607bbf8cd65b217fa845ff24245743c407bc5c8bbd260b5e"
May 25 23:35:28.160021 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:28.159998    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:29.003261 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.003226    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:29.003402 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.003271    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:29.003402 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.003241    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:29.003402 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:29.003367    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:29.003528 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:29.003431    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:29.003528 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:29.003481    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:29.100508 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.100483    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fnfhv_a5c0947b-069f-4202-bfff-2b66c658a943/dns-node-resolver/0.log"
May 25 23:35:29.148698 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.148607    2578 generic.go:358] "Generic (PLEG): container finished" podID="25656d22-e01a-49b8-b274-0514816aae9c" containerID="1ce14980047c0431d45b9b2a5234e0d6708f1b91cb61b3abdfa7f5285719d687" exitCode=0
May 25 23:35:29.149201 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.148688    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" event={"ID":"25656d22-e01a-49b8-b274-0514816aae9c","Type":"ContainerDied","Data":"1ce14980047c0431d45b9b2a5234e0d6708f1b91cb61b3abdfa7f5285719d687"}
May 25 23:35:29.153314 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.153292    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:35:29.153788 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.153765    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" event={"ID":"0ab4c691-103b-4095-a962-d3c780650959","Type":"ContainerStarted","Data":"4e42c2a280f7ac71d6bd182a181addf89e608f495929e3f27d10f5d40119d5cc"}
May 25 23:35:29.154200 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.154101    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:29.171480 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.171432    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:35:29.202310 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.202262    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2" podStartSLOduration=10.309498287 podStartE2EDuration="28.202244537s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:35:03.839170017 +0000 UTC m=+3.394114271" lastFinishedPulling="2026-05-25 23:35:21.731916264 +0000 UTC m=+21.286860521" observedRunningTime="2026-05-25 23:35:29.201585583 +0000 UTC m=+28.756529844" watchObservedRunningTime="2026-05-25 23:35:29.202244537 +0000 UTC m=+28.757188801"
May 25 23:35:29.397546 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.397513    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lwvj8"]
May 25 23:35:29.397712 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.397614    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:29.397783 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:29.397709    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:29.405281 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.405212    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9s245"]
May 25 23:35:29.405394 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.405314    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:29.405428 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:29.405402    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:29.414325 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.414288    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zmrpl"]
May 25 23:35:29.414440 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:29.414389    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:29.414496 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:29.414476    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:30.158021 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:30.157833    2578 generic.go:358] "Generic (PLEG): container finished" podID="25656d22-e01a-49b8-b274-0514816aae9c" containerID="90f6a98e8ac36b3bf84dd8351322644c9b35c051fd300ff63f84ee4079e5b0be" exitCode=0
May 25 23:35:30.158021 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:30.157892    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" event={"ID":"25656d22-e01a-49b8-b274-0514816aae9c","Type":"ContainerDied","Data":"90f6a98e8ac36b3bf84dd8351322644c9b35c051fd300ff63f84ee4079e5b0be"}
May 25 23:35:30.280829 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:30.280800    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qhm9s_5783a810-60b3-470b-a71e-98357f95b962/node-ca/0.log"
May 25 23:35:31.006623 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:31.006516    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:31.006805 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:31.006664    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:31.006805 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:31.006682    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:31.006915 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:31.006828    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:31.006915 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:31.006837    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:31.007080 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:31.006957    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:33.003763 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:33.003713    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:33.004420 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:33.003782    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:33.004420 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:33.003848    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:33.004420 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:33.003930    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:33.004420 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:33.004115    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:33.004420 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:33.004205    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:34.769586 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:34.769549    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:34.769586 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:34.769593    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:34.770251 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:34.769716    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:34.770251 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:34.769754    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 23:35:34.770251 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:34.769803    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs podName:accf65f4-6bda-4e39-ba44-c92d75f21e64 nodeName:}" failed. No retries permitted until 2026-05-25 23:36:06.769782758 +0000 UTC m=+66.324727018 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs") pod "network-metrics-daemon-lwvj8" (UID: "accf65f4-6bda-4e39-ba44-c92d75f21e64") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 23:35:34.770251 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:34.769836    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret podName:f0633bd5-0b65-4ce4-9fb5-963307feb3b6 nodeName:}" failed. No retries permitted until 2026-05-25 23:36:06.76982697 +0000 UTC m=+66.324771209 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret") pod "global-pull-secret-syncer-9s245" (UID: "f0633bd5-0b65-4ce4-9fb5-963307feb3b6") : object "kube-system"/"original-pull-secret" not registered
May 25 23:35:34.870643 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:34.870604    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:34.870843 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:34.870801    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 23:35:34.870843 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:34.870820    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 23:35:34.870843 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:34.870830    2578 projected.go:194] Error preparing data for projected volume kube-api-access-lhlvg for pod openshift-network-diagnostics/network-check-target-zmrpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:34.870983 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:34.870894    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg podName:d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d nodeName:}" failed. No retries permitted until 2026-05-25 23:36:06.870873211 +0000 UTC m=+66.425817465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhlvg" (UniqueName: "kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg") pod "network-check-target-zmrpl" (UID: "d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 23:35:35.003241 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:35.003067    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:35.003241 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:35.003068    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:35.003241 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:35.003206    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:35.003575 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:35.003079    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:35.003575 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:35.003301    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:35.003575 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:35.003364    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:37.006432 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:37.006398    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:37.007112 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:37.006398    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:37.007112 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:37.006514    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:37.007112 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:37.006398    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:37.007112 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:37.006567    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:37.007112 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:37.006650    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:37.174180 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:37.174146    2578 generic.go:358] "Generic (PLEG): container finished" podID="25656d22-e01a-49b8-b274-0514816aae9c" containerID="ad6d15aa187d831a6e717138bf5f961f7a2e9dd69cbf20316b4934ed585e9c0c" exitCode=0
May 25 23:35:37.174339 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:37.174194    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" event={"ID":"25656d22-e01a-49b8-b274-0514816aae9c","Type":"ContainerDied","Data":"ad6d15aa187d831a6e717138bf5f961f7a2e9dd69cbf20316b4934ed585e9c0c"}
May 25 23:35:38.179070 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:38.179037    2578 generic.go:358] "Generic (PLEG): container finished" podID="25656d22-e01a-49b8-b274-0514816aae9c" containerID="63d04dea41a1dd41a0105a300fea91bf7584b909fb4c9b2fbe1ec8905ff9c1b3" exitCode=0
May 25 23:35:38.179533 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:38.179108    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" event={"ID":"25656d22-e01a-49b8-b274-0514816aae9c","Type":"ContainerDied","Data":"63d04dea41a1dd41a0105a300fea91bf7584b909fb4c9b2fbe1ec8905ff9c1b3"}
May 25 23:35:39.003416 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:39.003225    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:39.003603 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:39.003244    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:39.003603 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:39.003488    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:39.003754 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:39.003596    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:39.003754 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:39.003244    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:39.003754 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:39.003702    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:39.183882 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:39.183798    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" event={"ID":"25656d22-e01a-49b8-b274-0514816aae9c","Type":"ContainerStarted","Data":"8b352a27e843e3efb417cf6e775a22851d7b7fe85738645c72542db5529e060b"}
May 25 23:35:39.214269 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:39.214215    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h2c7q" podStartSLOduration=5.840075153 podStartE2EDuration="38.214201359s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:35:03.846254547 +0000 UTC m=+3.401198790" lastFinishedPulling="2026-05-25 23:35:36.220380757 +0000 UTC m=+35.775324996" observedRunningTime="2026-05-25 23:35:39.213610894 +0000 UTC m=+38.768555154" watchObservedRunningTime="2026-05-25 23:35:39.214201359 +0000 UTC m=+38.769145627"
May 25 23:35:41.004145 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:41.004106    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:41.004628 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:41.004193    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:41.004628 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:41.004280    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:41.004628 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:41.004377    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:41.004628 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:41.004453    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:41.004628 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:41.004517    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:43.003494 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:43.003458    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:43.003960 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:43.003462    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:43.003960 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:43.003564    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:43.003960 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:43.003462    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:43.003960 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:43.003633    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:43.003960 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:43.003709    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:45.003495 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:45.003456    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:45.003981 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:45.003458    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:45.003981 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:45.003558    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:45.003981 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:45.003458    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:45.003981 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:45.003637    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:45.003981 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:45.003774    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:47.002949 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:47.002914    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:47.003396 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:47.002914    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:47.003396 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:47.003026    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:47.003396 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:47.002914    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:47.003396 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:47.003109    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:47.003396 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:47.003207    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:49.003444 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.003405    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:49.004018 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.003405    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:49.004018 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:49.003550    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s245" podUID="f0633bd5-0b65-4ce4-9fb5-963307feb3b6"
May 25 23:35:49.004018 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.003405    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:49.004018 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:49.003617    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmrpl" podUID="d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d"
May 25 23:35:49.004018 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:49.003673    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwvj8" podUID="accf65f4-6bda-4e39-ba44-c92d75f21e64"
May 25 23:35:49.745567 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.745490    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeReady"
May 25 23:35:49.745776 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.745625    2578 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 23:35:49.785293 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.785247    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-695546cc48-lqqmc"]
May 25 23:35:49.815995 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.815967    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4gjmv"]
May 25 23:35:49.816160 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.816130    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.818754 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.818701    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 23:35:49.818754 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.818701    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xdtnl\""
May 25 23:35:49.819048 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.819032    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 23:35:49.819273 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.819257    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 23:35:49.828502 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.828473    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 23:35:49.829941 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.829918    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-695546cc48-lqqmc"]
May 25 23:35:49.829941 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.829943    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t89ch"]
May 25 23:35:49.830119 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.830065    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.832544 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.832524    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 23:35:49.832813 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.832790    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 23:35:49.832922 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.832830    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 23:35:49.832922 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.832861    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lktvb\""
May 25 23:35:49.832922 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.832831    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 23:35:49.844180 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.844157    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4gjmv"]
May 25 23:35:49.844180 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.844182    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t89ch"]
May 25 23:35:49.844347 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.844307    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t89ch"
May 25 23:35:49.846933 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.846913    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d4kmw\""
May 25 23:35:49.847278 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.847259    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 23:35:49.847349 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.847279    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 23:35:49.847400 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.847381    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 23:35:49.882502 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5079703c-b039-467d-9bde-f1b9f5768bc5-trusted-ca\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.882502 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdm7\" (UniqueName: \"kubernetes.io/projected/eef71f49-b051-439d-91ef-3eb513154b1f-kube-api-access-qpdm7\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.882837 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8p97\" (UniqueName: \"kubernetes.io/projected/facb9575-57af-4a8d-8e9e-453d4ad52215-kube-api-access-j8p97\") pod \"ingress-canary-t89ch\" (UID: \"facb9575-57af-4a8d-8e9e-453d4ad52215\") " pod="openshift-ingress-canary/ingress-canary-t89ch"
May 25 23:35:49.882837 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5079703c-b039-467d-9bde-f1b9f5768bc5-image-registry-private-configuration\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.882837 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5079703c-b039-467d-9bde-f1b9f5768bc5-bound-sa-token\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.882837 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eef71f49-b051-439d-91ef-3eb513154b1f-crio-socket\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.882837 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5079703c-b039-467d-9bde-f1b9f5768bc5-ca-trust-extracted\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.882837 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5079703c-b039-467d-9bde-f1b9f5768bc5-registry-certificates\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.882837 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eef71f49-b051-439d-91ef-3eb513154b1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.883075 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/facb9575-57af-4a8d-8e9e-453d4ad52215-cert\") pod \"ingress-canary-t89ch\" (UID: \"facb9575-57af-4a8d-8e9e-453d4ad52215\") " pod="openshift-ingress-canary/ingress-canary-t89ch"
May 25 23:35:49.883075 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5079703c-b039-467d-9bde-f1b9f5768bc5-registry-tls\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.883075 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjq44\" (UniqueName: \"kubernetes.io/projected/5079703c-b039-467d-9bde-f1b9f5768bc5-kube-api-access-rjq44\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.883075 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5079703c-b039-467d-9bde-f1b9f5768bc5-installation-pull-secrets\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.883075 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eef71f49-b051-439d-91ef-3eb513154b1f-data-volume\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.883075 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.882981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eef71f49-b051-439d-91ef-3eb513154b1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.907472 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.907434    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wkpwv"]
May 25 23:35:49.923994 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.923962    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wkpwv"]
May 25 23:35:49.924150 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.924130    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:49.926697 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.926670    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 23:35:49.926883 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.926864    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 23:35:49.927020 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.927003    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4pndb\""
May 25 23:35:49.984244 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqht\" (UniqueName: \"kubernetes.io/projected/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-kube-api-access-ctqht\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:49.984393 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984281    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eef71f49-b051-439d-91ef-3eb513154b1f-data-volume\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.984393 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984307    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eef71f49-b051-439d-91ef-3eb513154b1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.984393 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-config-volume\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:49.984506 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984419    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5079703c-b039-467d-9bde-f1b9f5768bc5-trusted-ca\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.984506 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984453    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdm7\" (UniqueName: \"kubernetes.io/projected/eef71f49-b051-439d-91ef-3eb513154b1f-kube-api-access-qpdm7\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.984506 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984472    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j8p97\" (UniqueName: \"kubernetes.io/projected/facb9575-57af-4a8d-8e9e-453d4ad52215-kube-api-access-j8p97\") pod \"ingress-canary-t89ch\" (UID: \"facb9575-57af-4a8d-8e9e-453d4ad52215\") " pod="openshift-ingress-canary/ingress-canary-t89ch"
May 25 23:35:49.984506 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984490    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5079703c-b039-467d-9bde-f1b9f5768bc5-image-registry-private-configuration\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.984655 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984621    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eef71f49-b051-439d-91ef-3eb513154b1f-data-volume\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.984744 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984702    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5079703c-b039-467d-9bde-f1b9f5768bc5-bound-sa-token\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.984808 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984761    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eef71f49-b051-439d-91ef-3eb513154b1f-crio-socket\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.984808 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984796    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5079703c-b039-467d-9bde-f1b9f5768bc5-ca-trust-extracted\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.984892 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984821    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5079703c-b039-467d-9bde-f1b9f5768bc5-registry-certificates\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.984892 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-metrics-tls\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:49.984994 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984952    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eef71f49-b051-439d-91ef-3eb513154b1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.984994 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984982    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eef71f49-b051-439d-91ef-3eb513154b1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.985093 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.984999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-tmp-dir\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:49.985093 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.985029    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/facb9575-57af-4a8d-8e9e-453d4ad52215-cert\") pod \"ingress-canary-t89ch\" (UID: \"facb9575-57af-4a8d-8e9e-453d4ad52215\") " pod="openshift-ingress-canary/ingress-canary-t89ch"
May 25 23:35:49.985093 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.985057    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eef71f49-b051-439d-91ef-3eb513154b1f-crio-socket\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.985093 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.985068    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5079703c-b039-467d-9bde-f1b9f5768bc5-registry-tls\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.985285 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.985093    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rjq44\" (UniqueName: \"kubernetes.io/projected/5079703c-b039-467d-9bde-f1b9f5768bc5-kube-api-access-rjq44\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.985285 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.985124    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5079703c-b039-467d-9bde-f1b9f5768bc5-installation-pull-secrets\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.985285 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.985180    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5079703c-b039-467d-9bde-f1b9f5768bc5-ca-trust-extracted\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.985658 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.985628    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5079703c-b039-467d-9bde-f1b9f5768bc5-trusted-ca\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.985789 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.985677    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5079703c-b039-467d-9bde-f1b9f5768bc5-registry-certificates\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.988911 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.988888    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5079703c-b039-467d-9bde-f1b9f5768bc5-installation-pull-secrets\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.989014 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.988949    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eef71f49-b051-439d-91ef-3eb513154b1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.989062 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.989006    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5079703c-b039-467d-9bde-f1b9f5768bc5-registry-tls\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.989062 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.989041    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/facb9575-57af-4a8d-8e9e-453d4ad52215-cert\") pod \"ingress-canary-t89ch\" (UID: \"facb9575-57af-4a8d-8e9e-453d4ad52215\") " pod="openshift-ingress-canary/ingress-canary-t89ch"
May 25 23:35:49.989153 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.989134    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5079703c-b039-467d-9bde-f1b9f5768bc5-image-registry-private-configuration\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.993264 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.993236    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdm7\" (UniqueName: \"kubernetes.io/projected/eef71f49-b051-439d-91ef-3eb513154b1f-kube-api-access-qpdm7\") pod \"insights-runtime-extractor-4gjmv\" (UID: \"eef71f49-b051-439d-91ef-3eb513154b1f\") " pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:49.993909 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.993891    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5079703c-b039-467d-9bde-f1b9f5768bc5-bound-sa-token\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.994544 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.994522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjq44\" (UniqueName: \"kubernetes.io/projected/5079703c-b039-467d-9bde-f1b9f5768bc5-kube-api-access-rjq44\") pod \"image-registry-695546cc48-lqqmc\" (UID: \"5079703c-b039-467d-9bde-f1b9f5768bc5\") " pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:49.994614 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:49.994522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8p97\" (UniqueName: \"kubernetes.io/projected/facb9575-57af-4a8d-8e9e-453d4ad52215-kube-api-access-j8p97\") pod \"ingress-canary-t89ch\" (UID: \"facb9575-57af-4a8d-8e9e-453d4ad52215\") " pod="openshift-ingress-canary/ingress-canary-t89ch"
May 25 23:35:50.085846 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.085812    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqht\" (UniqueName: \"kubernetes.io/projected/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-kube-api-access-ctqht\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:50.086260 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.085867    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-config-volume\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:50.086260 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.085956    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-metrics-tls\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:50.086260 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.086002    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-tmp-dir\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:50.086428 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.086331    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-tmp-dir\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:50.086471 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.086430    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-config-volume\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:50.088296 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.088272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-metrics-tls\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:50.094313 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.094290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqht\" (UniqueName: \"kubernetes.io/projected/122dfc84-a6fe-4de1-9b59-7b4fe23c4d56-kube-api-access-ctqht\") pod \"dns-default-wkpwv\" (UID: \"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56\") " pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:50.127331 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.127296    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:50.139295 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.139256    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4gjmv"
May 25 23:35:50.153636 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.153603    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t89ch"
May 25 23:35:50.236181 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.233522    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:50.337996 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.337921    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-695546cc48-lqqmc"]
May 25 23:35:50.405903 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.405875    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wkpwv"]
May 25 23:35:50.409227 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:50.409198    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122dfc84_a6fe_4de1_9b59_7b4fe23c4d56.slice/crio-f4c97695dc8c7f79943aeeb64aae6624a548fd87baf61577b8947768893f4dc8 WatchSource:0}: Error finding container f4c97695dc8c7f79943aeeb64aae6624a548fd87baf61577b8947768893f4dc8: Status 404 returned error can't find the container with id f4c97695dc8c7f79943aeeb64aae6624a548fd87baf61577b8947768893f4dc8
May 25 23:35:50.551582 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.551545    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4gjmv"]
May 25 23:35:50.554241 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:50.554172    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t89ch"]
May 25 23:35:50.555676 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:50.555649    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef71f49_b051_439d_91ef_3eb513154b1f.slice/crio-377ef2671c5a2be0f9c36c95f9de1e5e3c501454b94bcf45653f50f135347ea8 WatchSource:0}: Error finding container 377ef2671c5a2be0f9c36c95f9de1e5e3c501454b94bcf45653f50f135347ea8: Status 404 returned error can't find the container with id 377ef2671c5a2be0f9c36c95f9de1e5e3c501454b94bcf45653f50f135347ea8
May 25 23:35:50.558463 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:50.558432    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacb9575_57af_4a8d_8e9e_453d4ad52215.slice/crio-e85ee3615e6b02c42dca5e2f0022d1ffe91ad0cea3c1749c8b5b259072a2210d WatchSource:0}: Error finding container e85ee3615e6b02c42dca5e2f0022d1ffe91ad0cea3c1749c8b5b259072a2210d: Status 404 returned error can't find the container with id e85ee3615e6b02c42dca5e2f0022d1ffe91ad0cea3c1749c8b5b259072a2210d
May 25 23:35:51.015505 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.015369    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:35:51.015505 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.015498    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:35:51.015804 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.015612    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:35:51.018309 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.018287    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 23:35:51.019440 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.019415    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 23:35:51.019545 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.019471    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 23:35:51.019545 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.019415    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hz8ds\""
May 25 23:35:51.019694 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.019428    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vfv7v\""
May 25 23:35:51.019694 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.019475    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 23:35:51.209937 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.209896    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t89ch" event={"ID":"facb9575-57af-4a8d-8e9e-453d4ad52215","Type":"ContainerStarted","Data":"e85ee3615e6b02c42dca5e2f0022d1ffe91ad0cea3c1749c8b5b259072a2210d"}
May 25 23:35:51.211828 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.211746    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-695546cc48-lqqmc" event={"ID":"5079703c-b039-467d-9bde-f1b9f5768bc5","Type":"ContainerStarted","Data":"ce95a3ca09120482d295e8a8e0eec2efbb1a4d440407368c88b039ec3e584b7e"}
May 25 23:35:51.211828 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.211784    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-695546cc48-lqqmc" event={"ID":"5079703c-b039-467d-9bde-f1b9f5768bc5","Type":"ContainerStarted","Data":"42f5e2c4b9c40ef28ab63e5ee5ebd38fb77ef8aacec74e23b8800119dfe8f874"}
May 25 23:35:51.212129 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.211907    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:35:51.212911 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.212878    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkpwv" event={"ID":"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56","Type":"ContainerStarted","Data":"f4c97695dc8c7f79943aeeb64aae6624a548fd87baf61577b8947768893f4dc8"}
May 25 23:35:51.214401 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.214374    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4gjmv" event={"ID":"eef71f49-b051-439d-91ef-3eb513154b1f","Type":"ContainerStarted","Data":"f31f30a785e6349cf7dfcf7e98a3c85416e54d6b5bc21f1b5c372034d02bbdd3"}
May 25 23:35:51.214520 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.214407    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4gjmv" event={"ID":"eef71f49-b051-439d-91ef-3eb513154b1f","Type":"ContainerStarted","Data":"377ef2671c5a2be0f9c36c95f9de1e5e3c501454b94bcf45653f50f135347ea8"}
May 25 23:35:51.234135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:51.234081    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-695546cc48-lqqmc" podStartSLOduration=3.234061468 podStartE2EDuration="3.234061468s" podCreationTimestamp="2026-05-25 23:35:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:35:51.232805089 +0000 UTC m=+50.787749349" watchObservedRunningTime="2026-05-25 23:35:51.234061468 +0000 UTC m=+50.789005752"
May 25 23:35:52.697676 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:52.697644    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"]
May 25 23:35:52.716278 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:52.716249    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"]
May 25 23:35:52.716420 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:52.716359    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"
May 25 23:35:52.718931 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:52.718907    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 25 23:35:52.719074 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:52.718911    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-phjfj\""
May 25 23:35:52.815446 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:52.815409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-prbm4\" (UID: \"4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"
May 25 23:35:52.916614 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:52.916577    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-prbm4\" (UID: \"4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"
May 25 23:35:52.916770 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:52.916744    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 25 23:35:52.916826 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:35:52.916816    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4-tls-certificates podName:4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4 nodeName:}" failed. No retries permitted until 2026-05-25 23:35:53.416799708 +0000 UTC m=+52.971743947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-prbm4" (UID: "4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4") : secret "prometheus-operator-admission-webhook-tls" not found
May 25 23:35:53.420802 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:53.420776    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-prbm4\" (UID: \"4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"
May 25 23:35:53.423213 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:53.423183    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-prbm4\" (UID: \"4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"
May 25 23:35:53.625743 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:53.625630    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"
May 25 23:35:53.752507 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:53.752476    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"]
May 25 23:35:53.755544 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:35:53.755509    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5d7cf7_ba46_4e2c_a437_7883d5c7fea4.slice/crio-97eed0f61d43632c39c7dba38f40cd9bd6f94737995c503c34948995f58521e0 WatchSource:0}: Error finding container 97eed0f61d43632c39c7dba38f40cd9bd6f94737995c503c34948995f58521e0: Status 404 returned error can't find the container with id 97eed0f61d43632c39c7dba38f40cd9bd6f94737995c503c34948995f58521e0
May 25 23:35:54.224355 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:54.224299    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t89ch" event={"ID":"facb9575-57af-4a8d-8e9e-453d4ad52215","Type":"ContainerStarted","Data":"01e55f669b3729b3adb2193aa89a8be495b1028c72ecb313d3e510aa17320f02"}
May 25 23:35:54.227848 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:54.226907    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkpwv" event={"ID":"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56","Type":"ContainerStarted","Data":"0b836f2b64346fec9954fdbfbf7879593a9e10ead5bbd1d13d154dc512d13c8d"}
May 25 23:35:54.227848 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:54.226944    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkpwv" event={"ID":"122dfc84-a6fe-4de1-9b59-7b4fe23c4d56","Type":"ContainerStarted","Data":"79df6966b61da537d1d518d743d7ce584ad7adf5253e065fce7b6fc1815e4cd1"}
May 25 23:35:54.227848 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:54.227400    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wkpwv"
May 25 23:35:54.229369 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:54.229342    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4gjmv" event={"ID":"eef71f49-b051-439d-91ef-3eb513154b1f","Type":"ContainerStarted","Data":"9f6d64d3508b50d7bab6af14eff6cc64e3f2ced444beeffc72b79605cd820508"}
May 25 23:35:54.230447 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:54.230422    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4" event={"ID":"4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4","Type":"ContainerStarted","Data":"97eed0f61d43632c39c7dba38f40cd9bd6f94737995c503c34948995f58521e0"}
May 25 23:35:54.242524 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:54.242379    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t89ch" podStartSLOduration=2.700408817 podStartE2EDuration="5.242364937s" podCreationTimestamp="2026-05-25 23:35:49 +0000 UTC" firstStartedPulling="2026-05-25 23:35:50.56033447 +0000 UTC m=+50.115278709" lastFinishedPulling="2026-05-25 23:35:53.10229059 +0000 UTC m=+52.657234829" observedRunningTime="2026-05-25 23:35:54.2420778 +0000 UTC m=+53.797022063" watchObservedRunningTime="2026-05-25 23:35:54.242364937 +0000 UTC m=+53.797309199"
May 25 23:35:54.261432 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:54.261379    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-wkpwv" podStartSLOduration=2.574967488 podStartE2EDuration="5.261359812s" podCreationTimestamp="2026-05-25 23:35:49 +0000 UTC" firstStartedPulling="2026-05-25 23:35:50.411363462 +0000 UTC m=+49.966307718" lastFinishedPulling="2026-05-25 23:35:53.097755797 +0000 UTC m=+52.652700042" observedRunningTime="2026-05-25 23:35:54.260737428 +0000 UTC m=+53.815681687" watchObservedRunningTime="2026-05-25 23:35:54.261359812 +0000 UTC m=+53.816304075"
May 25 23:35:56.236598 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:56.236565    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4gjmv" event={"ID":"eef71f49-b051-439d-91ef-3eb513154b1f","Type":"ContainerStarted","Data":"80d94f83cc81af04a0190fa94465a03f2824bd4cd2c07e2ac7897301c16f1b3f"}
May 25 23:35:56.237948 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:56.237916    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4" event={"ID":"4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4","Type":"ContainerStarted","Data":"0a46a23a45a42ac214e3695ceb1b1df2d7b008ffb0401744f9935419ef606f00"}
May 25 23:35:56.238121 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:56.238106    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"
May 25 23:35:56.242639 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:56.242620    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4"
May 25 23:35:56.262261 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:56.262211    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4gjmv" podStartSLOduration=2.24408122 podStartE2EDuration="7.262196165s" podCreationTimestamp="2026-05-25 23:35:49 +0000 UTC" firstStartedPulling="2026-05-25 23:35:50.643679274 +0000 UTC m=+50.198623513" lastFinishedPulling="2026-05-25 23:35:55.661794205 +0000 UTC m=+55.216738458" observedRunningTime="2026-05-25 23:35:56.261280301 +0000 UTC m=+55.816224553" watchObservedRunningTime="2026-05-25 23:35:56.262196165 +0000 UTC m=+55.817140426"
May 25 23:35:56.277993 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:35:56.277942    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-prbm4" podStartSLOduration=2.371778266 podStartE2EDuration="4.277927874s" podCreationTimestamp="2026-05-25 23:35:52 +0000 UTC" firstStartedPulling="2026-05-25 23:35:53.757470038 +0000 UTC m=+53.312414277" lastFinishedPulling="2026-05-25 23:35:55.663619638 +0000 UTC m=+55.218563885" observedRunningTime="2026-05-25 23:35:56.276996885 +0000 UTC m=+55.831941142" watchObservedRunningTime="2026-05-25 23:35:56.277927874 +0000 UTC m=+55.832872135"
May 25 23:36:01.171456 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:01.171422    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9d2"
May 25 23:36:02.200712 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.200672    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nq27j"]
May 25 23:36:02.203511 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.203495    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.206568 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.206543    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 23:36:02.206717 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.206586    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 23:36:02.206717 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.206629    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 23:36:02.207680 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.207659    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 23:36:02.207761 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.207744    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 23:36:02.207826 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.207765    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 23:36:02.207826 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.207812    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qnf7k\""
May 25 23:36:02.282052 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.282011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-wtmp\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.282229 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.282071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-metrics-client-ca\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.282229 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.282131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-textfile\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.282229 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.282157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.282229 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.282179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4cxj\" (UniqueName: \"kubernetes.io/projected/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-kube-api-access-x4cxj\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.282229 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.282215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-sys\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.282229 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.282233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-tls\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.282450 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.282252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-accelerators-collector-config\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.282450 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.282271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-root\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.383600 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.383560    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-tls\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.383798 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.383600    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-accelerators-collector-config\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.383798 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.383648    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-root\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.383798 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.383694    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-wtmp\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.383798 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.383711    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-metrics-client-ca\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.383798 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.383772    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-root\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.384055 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.383860    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-wtmp\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.384055 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.383868    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-textfile\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.384055 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.383983    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.384055 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.384016    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x4cxj\" (UniqueName: \"kubernetes.io/projected/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-kube-api-access-x4cxj\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.384255 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.384071    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-sys\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.384255 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.384145    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-sys\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.384255 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.384169    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-textfile\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.384404 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.384346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-metrics-client-ca\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.384453 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.384397    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-accelerators-collector-config\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.386016 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.386000    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-tls\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.386217 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.386201    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.392546 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.392525    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4cxj\" (UniqueName: \"kubernetes.io/projected/4b3d5441-69ab-4f3d-ac9f-4b2076699d8c-kube-api-access-x4cxj\") pod \"node-exporter-nq27j\" (UID: \"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c\") " pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.513668 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:02.513585    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nq27j"
May 25 23:36:02.521189 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:36:02.521162    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3d5441_69ab_4f3d_ac9f_4b2076699d8c.slice/crio-c8310fd3b96f9532ba1185bfb355dff7bfacb277f741e4354a35a6292ed17566 WatchSource:0}: Error finding container c8310fd3b96f9532ba1185bfb355dff7bfacb277f741e4354a35a6292ed17566: Status 404 returned error can't find the container with id c8310fd3b96f9532ba1185bfb355dff7bfacb277f741e4354a35a6292ed17566
May 25 23:36:03.218275 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.218243    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 23:36:03.225539 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.225509    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.228304 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.228275    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8hnsp\""
May 25 23:36:03.228447 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.228324    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 23:36:03.228447 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.228275    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 23:36:03.228447 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.228399    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 23:36:03.228447 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.228327    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 23:36:03.228685 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.228637    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 23:36:03.228778 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.228757    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 23:36:03.228881 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.228776    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 23:36:03.229084 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.229065    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 23:36:03.229662 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.229643    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 23:36:03.239685 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.239653    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 23:36:03.258281 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.258187    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nq27j" event={"ID":"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c","Type":"ContainerStarted","Data":"c8310fd3b96f9532ba1185bfb355dff7bfacb277f741e4354a35a6292ed17566"}
May 25 23:36:03.290509 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.290713 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.290713 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.290713 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.290713 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.290713 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.290713 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-web-config\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.291025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.291025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-config-volume\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.291025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.291025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cv5b\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-kube-api-access-6cv5b\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.291025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-config-out\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.291025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.290970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.391644 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.391608    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-config-out\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.391644 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.391650    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.391909 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.391715    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.391909 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.391764    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.392022 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.391917    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.392022 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.391955    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.392142 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.392119    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.392217 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.392167    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.392264 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.392215    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-web-config\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.392264 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.392243    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.392354 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.392279    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-config-volume\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.392354 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.392317    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.392354 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.392341    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6cv5b\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-kube-api-access-6cv5b\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.393436 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.393405    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.394654 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.394625    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-config-out\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.394974 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.394946    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.395480 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.395452    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.395639 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:36:03.395621    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-alertmanager-trusted-ca-bundle podName:32db75f5-a85b-4778-91fd-599684318068 nodeName:}" failed. No retries permitted until 2026-05-25 23:36:03.895598605 +0000 UTC m=+63.450542861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "32db75f5-a85b-4778-91fd-599684318068") : configmap references non-existent config key: ca-bundle.crt
May 25 23:36:03.396013 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.395988    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-config-volume\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.396119 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.396062    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.396301 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.396279    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.396506 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.396481    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.396642 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.396612    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.397190 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.397142    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-web-config\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.398295 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.398273    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.402182 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.402158    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cv5b\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-kube-api-access-6cv5b\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.896567 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.896475    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:03.897385 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:03.897364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:04.137125 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.137088    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 23:36:04.232799 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.232768    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"]
May 25 23:36:04.235764 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.235702    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wkpwv"
May 25 23:36:04.235929 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.235861    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.239462 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.239245    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 25 23:36:04.239462 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.239268    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 25 23:36:04.239462 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.239298    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 25 23:36:04.239462 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.239312    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-njztl\""
May 25 23:36:04.239462 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.239298    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 25 23:36:04.239462 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.239317    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 25 23:36:04.239462 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.239366    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-29figd57bf592\""
May 25 23:36:04.250139 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.250105    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"]
May 25 23:36:04.262049 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.262018    2578 generic.go:358] "Generic (PLEG): container finished" podID="4b3d5441-69ab-4f3d-ac9f-4b2076699d8c" containerID="346a1346d175fe410df71c3d00ad66d9d3a8c5ffa0601325f54c50bfd29fb002" exitCode=0
May 25 23:36:04.262207 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.262081    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nq27j" event={"ID":"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c","Type":"ContainerDied","Data":"346a1346d175fe410df71c3d00ad66d9d3a8c5ffa0601325f54c50bfd29fb002"}
May 25 23:36:04.276314 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.276288    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 23:36:04.279605 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:36:04.279573    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32db75f5_a85b_4778_91fd_599684318068.slice/crio-7ad5997e5e9893395c553252fe45d7c46c1765d00fd96b82f9c5ae2c169b73a3 WatchSource:0}: Error finding container 7ad5997e5e9893395c553252fe45d7c46c1765d00fd96b82f9c5ae2c169b73a3: Status 404 returned error can't find the container with id 7ad5997e5e9893395c553252fe45d7c46c1765d00fd96b82f9c5ae2c169b73a3
May 25 23:36:04.300395 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.300363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca119016-8532-4763-a583-37314c63f894-metrics-client-ca\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.300586 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.300434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2jb\" (UniqueName: \"kubernetes.io/projected/ca119016-8532-4763-a583-37314c63f894-kube-api-access-pf2jb\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.300586 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.300512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.300698 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.300664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.300783 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.300767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.300921 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.300834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-grpc-tls\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.300921 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.300885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.301066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.300937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-tls\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.402275 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.402235    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.402501 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.402343    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-tls\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.402640 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.402608    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca119016-8532-4763-a583-37314c63f894-metrics-client-ca\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.403239 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.403216    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2jb\" (UniqueName: \"kubernetes.io/projected/ca119016-8532-4763-a583-37314c63f894-kube-api-access-pf2jb\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.403357 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.403272    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.403357 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.403318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca119016-8532-4763-a583-37314c63f894-metrics-client-ca\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.403357 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.403338    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.403507 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.403436    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.403507 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.403468    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-grpc-tls\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.405302 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.405254    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.405977 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.405933    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-tls\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.406209 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.406162    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.406390 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.406366    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.406846 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.406827    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-grpc-tls\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.406944 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.406930    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca119016-8532-4763-a583-37314c63f894-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.411329 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.411304    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2jb\" (UniqueName: \"kubernetes.io/projected/ca119016-8532-4763-a583-37314c63f894-kube-api-access-pf2jb\") pod \"thanos-querier-57c58f7bb8-xvqmp\" (UID: \"ca119016-8532-4763-a583-37314c63f894\") " pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.546459 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.546356    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:04.681490 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:04.681464    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"]
May 25 23:36:04.683686 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:36:04.683659    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca119016_8532_4763_a583_37314c63f894.slice/crio-2597d980ee31041dc2fd337dbeeebf4ec972b31c48199e6585888ead3246e8ed WatchSource:0}: Error finding container 2597d980ee31041dc2fd337dbeeebf4ec972b31c48199e6585888ead3246e8ed: Status 404 returned error can't find the container with id 2597d980ee31041dc2fd337dbeeebf4ec972b31c48199e6585888ead3246e8ed
May 25 23:36:05.266351 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:05.266318    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerStarted","Data":"e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c"}
May 25 23:36:05.266718 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:05.266363    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerStarted","Data":"7ad5997e5e9893395c553252fe45d7c46c1765d00fd96b82f9c5ae2c169b73a3"}
May 25 23:36:05.267491 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:05.267466    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp" event={"ID":"ca119016-8532-4763-a583-37314c63f894","Type":"ContainerStarted","Data":"2597d980ee31041dc2fd337dbeeebf4ec972b31c48199e6585888ead3246e8ed"}
May 25 23:36:05.269496 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:05.269471    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nq27j" event={"ID":"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c","Type":"ContainerStarted","Data":"0fd4218a2f4c73b83dd344e8f5b55a237a3da2ca754a0c57aec6a7dbfbf40263"}
May 25 23:36:05.269612 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:05.269502    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nq27j" event={"ID":"4b3d5441-69ab-4f3d-ac9f-4b2076699d8c","Type":"ContainerStarted","Data":"d6fefd26193a8a543f5e0fb5eb27359349d848d2e4e67dde62a74b34ede2ae54"}
May 25 23:36:05.301575 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:05.301528    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nq27j" podStartSLOduration=2.251322022 podStartE2EDuration="3.301512848s" podCreationTimestamp="2026-05-25 23:36:02 +0000 UTC" firstStartedPulling="2026-05-25 23:36:02.52293824 +0000 UTC m=+62.077882479" lastFinishedPulling="2026-05-25 23:36:03.573129061 +0000 UTC m=+63.128073305" observedRunningTime="2026-05-25 23:36:05.301291803 +0000 UTC m=+64.856236076" watchObservedRunningTime="2026-05-25 23:36:05.301512848 +0000 UTC m=+64.856457102"
May 25 23:36:06.274816 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.274780    2578 generic.go:358] "Generic (PLEG): container finished" podID="32db75f5-a85b-4778-91fd-599684318068" containerID="e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c" exitCode=0
May 25 23:36:06.275268 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.274871    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerDied","Data":"e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c"}
May 25 23:36:06.556949 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.556905    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"]
May 25 23:36:06.587063 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.587030    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"]
May 25 23:36:06.587266 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.587183    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.591025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.590829    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9jn5q\""
May 25 23:36:06.591025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.590838    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 25 23:36:06.591025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.590889    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 25 23:36:06.591025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.590890    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9a7v9j3a57cku\""
May 25 23:36:06.591025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.590913    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 25 23:36:06.591377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.591246    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 25 23:36:06.724162 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.724120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e0ceea22-b652-4c2b-ac89-f728cef64d58-secret-metrics-server-client-certs\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.724162 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.724164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hrl2\" (UniqueName: \"kubernetes.io/projected/e0ceea22-b652-4c2b-ac89-f728cef64d58-kube-api-access-8hrl2\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.724389 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.724206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e0ceea22-b652-4c2b-ac89-f728cef64d58-secret-metrics-server-tls\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.724389 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.724239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0ceea22-b652-4c2b-ac89-f728cef64d58-client-ca-bundle\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.724389 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.724306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e0ceea22-b652-4c2b-ac89-f728cef64d58-metrics-server-audit-profiles\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.724389 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.724340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ceea22-b652-4c2b-ac89-f728cef64d58-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.724389 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.724368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e0ceea22-b652-4c2b-ac89-f728cef64d58-audit-log\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.825450 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.825356    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrl2\" (UniqueName: \"kubernetes.io/projected/e0ceea22-b652-4c2b-ac89-f728cef64d58-kube-api-access-8hrl2\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.825450 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.825407    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e0ceea22-b652-4c2b-ac89-f728cef64d58-secret-metrics-server-tls\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.825450 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.825439    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0ceea22-b652-4c2b-ac89-f728cef64d58-client-ca-bundle\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.825677 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.825507    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e0ceea22-b652-4c2b-ac89-f728cef64d58-metrics-server-audit-profiles\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.825677 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.825664    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ceea22-b652-4c2b-ac89-f728cef64d58-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.825784 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.825699    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e0ceea22-b652-4c2b-ac89-f728cef64d58-audit-log\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.826259 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.825826    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:36:06.826259 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.825871    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:36:06.826259 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.825907    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e0ceea22-b652-4c2b-ac89-f728cef64d58-secret-metrics-server-client-certs\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.826259 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.826180    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e0ceea22-b652-4c2b-ac89-f728cef64d58-audit-log\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.826517 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.826451    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0ceea22-b652-4c2b-ac89-f728cef64d58-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.826829 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.826808    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e0ceea22-b652-4c2b-ac89-f728cef64d58-metrics-server-audit-profiles\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.828453 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.828408    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e0ceea22-b652-4c2b-ac89-f728cef64d58-secret-metrics-server-tls\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.828534 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.828474    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e0ceea22-b652-4c2b-ac89-f728cef64d58-secret-metrics-server-client-certs\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.828583 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.828556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0ceea22-b652-4c2b-ac89-f728cef64d58-client-ca-bundle\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.828622 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.828599    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 23:36:06.828654 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.828634    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 23:36:06.833796 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.833777    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrl2\" (UniqueName: \"kubernetes.io/projected/e0ceea22-b652-4c2b-ac89-f728cef64d58-kube-api-access-8hrl2\") pod \"metrics-server-6dfc7747c8-kxcqg\" (UID: \"e0ceea22-b652-4c2b-ac89-f728cef64d58\") " pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.838490 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.838456    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0633bd5-0b65-4ce4-9fb5-963307feb3b6-original-pull-secret\") pod \"global-pull-secret-syncer-9s245\" (UID: \"f0633bd5-0b65-4ce4-9fb5-963307feb3b6\") " pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:36:06.838490 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.838472    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accf65f4-6bda-4e39-ba44-c92d75f21e64-metrics-certs\") pod \"network-metrics-daemon-lwvj8\" (UID: \"accf65f4-6bda-4e39-ba44-c92d75f21e64\") " pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:36:06.897098 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.897068    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:06.927091 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.927060    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:36:06.930169 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.930134    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 23:36:06.930472 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.930456    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hz8ds\""
May 25 23:36:06.938587 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.938554    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwvj8"
May 25 23:36:06.940631 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.940613    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 23:36:06.945019 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.945001    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s245"
May 25 23:36:06.950985 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:06.950959    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhlvg\" (UniqueName: \"kubernetes.io/projected/d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d-kube-api-access-lhlvg\") pod \"network-check-target-zmrpl\" (UID: \"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d\") " pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:36:07.182706 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.182674    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"]
May 25 23:36:07.186715 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:36:07.186688    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ceea22_b652_4c2b_ac89_f728cef64d58.slice/crio-308431b8b45a725c2142e981ba7cd6b801a4b2550ade22209306080a628604d6 WatchSource:0}: Error finding container 308431b8b45a725c2142e981ba7cd6b801a4b2550ade22209306080a628604d6: Status 404 returned error can't find the container with id 308431b8b45a725c2142e981ba7cd6b801a4b2550ade22209306080a628604d6
May 25 23:36:07.195496 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.195466    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9s245"]
May 25 23:36:07.201377 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:36:07.201341    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0633bd5_0b65_4ce4_9fb5_963307feb3b6.slice/crio-d836f5ed15f414ab5d3a1192b250d7894330725dfb92aae3069e70a6d58dfbfc WatchSource:0}: Error finding container d836f5ed15f414ab5d3a1192b250d7894330725dfb92aae3069e70a6d58dfbfc: Status 404 returned error can't find the container with id d836f5ed15f414ab5d3a1192b250d7894330725dfb92aae3069e70a6d58dfbfc
May 25 23:36:07.239901 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.239880    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vfv7v\""
May 25 23:36:07.247663 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.247634    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:36:07.279676 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.279453    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9s245" event={"ID":"f0633bd5-0b65-4ce4-9fb5-963307feb3b6","Type":"ContainerStarted","Data":"d836f5ed15f414ab5d3a1192b250d7894330725dfb92aae3069e70a6d58dfbfc"}
May 25 23:36:07.280770 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.280743    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg" event={"ID":"e0ceea22-b652-4c2b-ac89-f728cef64d58","Type":"ContainerStarted","Data":"308431b8b45a725c2142e981ba7cd6b801a4b2550ade22209306080a628604d6"}
May 25 23:36:07.282555 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.282533    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp" event={"ID":"ca119016-8532-4763-a583-37314c63f894","Type":"ContainerStarted","Data":"d38c1651c5460ebd6b793760cf0bb0823d1df3402752487daa8a1dafd1dc1e1f"}
May 25 23:36:07.381708 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.381131    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6846f6469c-gd6v8"]
May 25 23:36:07.404269 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.404129    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6846f6469c-gd6v8"]
May 25 23:36:07.404572 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.404320    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.407716 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.407380    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 25 23:36:07.407716 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.407451    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 25 23:36:07.407716 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.407520    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rsp6l\""
May 25 23:36:07.407716 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.407666    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 25 23:36:07.408087 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.407924    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lwvj8"]
May 25 23:36:07.408087 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.407988    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 25 23:36:07.408087 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.408023    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 25 23:36:07.412999 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.412974    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 25 23:36:07.413998 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:36:07.413970    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccf65f4_6bda_4e39_ba44_c92d75f21e64.slice/crio-63ee4949e52a3d35a2fb8564d24d552a594b36b6d628f6e684efcbe2f9866e03 WatchSource:0}: Error finding container 63ee4949e52a3d35a2fb8564d24d552a594b36b6d628f6e684efcbe2f9866e03: Status 404 returned error can't find the container with id 63ee4949e52a3d35a2fb8564d24d552a594b36b6d628f6e684efcbe2f9866e03
May 25 23:36:07.434616 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.434584    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zmrpl"]
May 25 23:36:07.438201 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:36:07.438170    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f3d6c8_2003_4ce2_9c0d_4ae0bb00a94d.slice/crio-4076b0b17e17e5f75501a6463a5c03de7d3c8a15921290b6948d8fdb0f6b0499 WatchSource:0}: Error finding container 4076b0b17e17e5f75501a6463a5c03de7d3c8a15921290b6948d8fdb0f6b0499: Status 404 returned error can't find the container with id 4076b0b17e17e5f75501a6463a5c03de7d3c8a15921290b6948d8fdb0f6b0499
May 25 23:36:07.534575 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.534535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63f26bc5-73a4-4b71-833a-c009538f5068-metrics-client-ca\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.534770 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.534586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-federate-client-tls\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.534770 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.534618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-secret-telemeter-client\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.534770 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.534754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-telemeter-client-tls\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.534921 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.534836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.534921 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.534882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f26bc5-73a4-4b71-833a-c009538f5068-serving-certs-ca-bundle\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.535069 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.534920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f26bc5-73a4-4b71-833a-c009538f5068-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.535069 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.534947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8svb\" (UniqueName: \"kubernetes.io/projected/63f26bc5-73a4-4b71-833a-c009538f5068-kube-api-access-g8svb\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.635879 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.635801    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-federate-client-tls\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.635879 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.635848    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-secret-telemeter-client\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.636121 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.635886    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-telemeter-client-tls\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.636121 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.635942    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.636121 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.635991    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f26bc5-73a4-4b71-833a-c009538f5068-serving-certs-ca-bundle\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.636121 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.636022    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f26bc5-73a4-4b71-833a-c009538f5068-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.636121 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.636045    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g8svb\" (UniqueName: \"kubernetes.io/projected/63f26bc5-73a4-4b71-833a-c009538f5068-kube-api-access-g8svb\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.636121 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.636092    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63f26bc5-73a4-4b71-833a-c009538f5068-metrics-client-ca\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.636935 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.636907    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63f26bc5-73a4-4b71-833a-c009538f5068-metrics-client-ca\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.637174 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.637146    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f26bc5-73a4-4b71-833a-c009538f5068-serving-certs-ca-bundle\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.637273 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.637193    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f26bc5-73a4-4b71-833a-c009538f5068-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.638492 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.638464    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-federate-client-tls\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.638593 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.638554    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-secret-telemeter-client\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.638593 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.638555    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.638771 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.638743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/63f26bc5-73a4-4b71-833a-c009538f5068-telemeter-client-tls\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.647849 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.647823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8svb\" (UniqueName: \"kubernetes.io/projected/63f26bc5-73a4-4b71-833a-c009538f5068-kube-api-access-g8svb\") pod \"telemeter-client-6846f6469c-gd6v8\" (UID: \"63f26bc5-73a4-4b71-833a-c009538f5068\") " pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:07.724844 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:07.724805    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8"
May 25 23:36:08.282142 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:08.281822    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6846f6469c-gd6v8"]
May 25 23:36:08.287335 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:08.287299    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zmrpl" event={"ID":"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d","Type":"ContainerStarted","Data":"4076b0b17e17e5f75501a6463a5c03de7d3c8a15921290b6948d8fdb0f6b0499"}
May 25 23:36:08.288464 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:08.288435    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lwvj8" event={"ID":"accf65f4-6bda-4e39-ba44-c92d75f21e64","Type":"ContainerStarted","Data":"63ee4949e52a3d35a2fb8564d24d552a594b36b6d628f6e684efcbe2f9866e03"}
May 25 23:36:08.291165 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:08.291141    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp" event={"ID":"ca119016-8532-4763-a583-37314c63f894","Type":"ContainerStarted","Data":"b4f385aaa585fa0a25ea07062cd4549e743f2020ded577480b7cbebf0d2ca8a0"}
May 25 23:36:08.291295 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:08.291186    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp" event={"ID":"ca119016-8532-4763-a583-37314c63f894","Type":"ContainerStarted","Data":"af83be051a0385848667c21eda9c8a3eca2c1e684965d9b224652ed603c484c5"}
May 25 23:36:08.474510 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:36:08.474426    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f26bc5_73a4_4b71_833a_c009538f5068.slice/crio-ccb4b3e8715463b102ca43d146da7c331dbe9d55a0445498629e76b989268197 WatchSource:0}: Error finding container ccb4b3e8715463b102ca43d146da7c331dbe9d55a0445498629e76b989268197: Status 404 returned error can't find the container with id ccb4b3e8715463b102ca43d146da7c331dbe9d55a0445498629e76b989268197
May 25 23:36:09.297374 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:09.297275    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8" event={"ID":"63f26bc5-73a4-4b71-833a-c009538f5068","Type":"ContainerStarted","Data":"ccb4b3e8715463b102ca43d146da7c331dbe9d55a0445498629e76b989268197"}
May 25 23:36:09.299713 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:09.299674    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerStarted","Data":"42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62"}
May 25 23:36:10.134679 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.134311    2578 patch_prober.go:28] interesting pod/image-registry-695546cc48-lqqmc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 23:36:10.134679 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.134421    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-695546cc48-lqqmc" podUID="5079703c-b039-467d-9bde-f1b9f5768bc5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 23:36:10.306742 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.305970    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg" event={"ID":"e0ceea22-b652-4c2b-ac89-f728cef64d58","Type":"ContainerStarted","Data":"ba1bc06d1a2bcdec51d95d1a8bd1c1cf723b01902d9c97a0d2832ea24ae65b97"}
May 25 23:36:10.314671 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.314641    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerStarted","Data":"2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9"}
May 25 23:36:10.314820 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.314682    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerStarted","Data":"7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8"}
May 25 23:36:10.314820 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.314695    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerStarted","Data":"8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027"}
May 25 23:36:10.314820 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.314710    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerStarted","Data":"25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8"}
May 25 23:36:10.318174 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.317538    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lwvj8" event={"ID":"accf65f4-6bda-4e39-ba44-c92d75f21e64","Type":"ContainerStarted","Data":"d8c945089804d000d398959a277e524ab404b0789f87b55270a67e5f7dd7b142"}
May 25 23:36:10.318174 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.317575    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lwvj8" event={"ID":"accf65f4-6bda-4e39-ba44-c92d75f21e64","Type":"ContainerStarted","Data":"7af9057a6b7a7b038ee279337b514c634c696e5b95b26ba4370685cf4e051690"}
May 25 23:36:10.328148 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.328059    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg" podStartSLOduration=1.686739472 podStartE2EDuration="4.328040526s" podCreationTimestamp="2026-05-25 23:36:06 +0000 UTC" firstStartedPulling="2026-05-25 23:36:07.189370853 +0000 UTC m=+66.744315096" lastFinishedPulling="2026-05-25 23:36:09.8306719 +0000 UTC m=+69.385616150" observedRunningTime="2026-05-25 23:36:10.327290784 +0000 UTC m=+69.882235048" watchObservedRunningTime="2026-05-25 23:36:10.328040526 +0000 UTC m=+69.882984787"
May 25 23:36:10.329479 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.329444    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp" event={"ID":"ca119016-8532-4763-a583-37314c63f894","Type":"ContainerStarted","Data":"6ebed9baeb9ac820a0024e76ce9d71066cbf70a85a017978c36fd652e1492b02"}
May 25 23:36:10.329579 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.329488    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp" event={"ID":"ca119016-8532-4763-a583-37314c63f894","Type":"ContainerStarted","Data":"6254ba1cd537ed33eed285b2a87adee214e732ce979821ce8174d5a390fc9806"}
May 25 23:36:10.329579 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.329507    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp" event={"ID":"ca119016-8532-4763-a583-37314c63f894","Type":"ContainerStarted","Data":"e4a02fd2304529ad4d2573ad364078b83a56ccab23dae383d7d3021dcd786929"}
May 25 23:36:10.330114 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.330091    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:10.344310 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.343956    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lwvj8" podStartSLOduration=66.925405535 podStartE2EDuration="1m9.343937074s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:36:07.416267053 +0000 UTC m=+66.971211308" lastFinishedPulling="2026-05-25 23:36:09.834798607 +0000 UTC m=+69.389742847" observedRunningTime="2026-05-25 23:36:10.34303127 +0000 UTC m=+69.897975535" watchObservedRunningTime="2026-05-25 23:36:10.343937074 +0000 UTC m=+69.898881336"
May 25 23:36:10.372078 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:10.371590    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp" podStartSLOduration=1.225154735 podStartE2EDuration="6.371568243s" podCreationTimestamp="2026-05-25 23:36:04 +0000 UTC" firstStartedPulling="2026-05-25 23:36:04.685603907 +0000 UTC m=+64.240548145" lastFinishedPulling="2026-05-25 23:36:09.832017414 +0000 UTC m=+69.386961653" observedRunningTime="2026-05-25 23:36:10.369440853 +0000 UTC m=+69.924385117" watchObservedRunningTime="2026-05-25 23:36:10.371568243 +0000 UTC m=+69.926512506"
May 25 23:36:11.336352 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:11.336295    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerStarted","Data":"b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b"}
May 25 23:36:11.366703 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:11.366644    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.142972128 podStartE2EDuration="8.366627323s" podCreationTimestamp="2026-05-25 23:36:03 +0000 UTC" firstStartedPulling="2026-05-25 23:36:04.281869378 +0000 UTC m=+63.836813632" lastFinishedPulling="2026-05-25 23:36:08.505524572 +0000 UTC m=+68.060468827" observedRunningTime="2026-05-25 23:36:11.364678473 +0000 UTC m=+70.919622734" watchObservedRunningTime="2026-05-25 23:36:11.366627323 +0000 UTC m=+70.921571583"
May 25 23:36:12.221505 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:12.221475    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-695546cc48-lqqmc"
May 25 23:36:14.352358 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:14.352312    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9s245" event={"ID":"f0633bd5-0b65-4ce4-9fb5-963307feb3b6","Type":"ContainerStarted","Data":"473d3494d3b5f87c9a30545b506314a9d3ce829508cb4b5bd8d035d72aa73c40"}
May 25 23:36:14.353782 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:14.353755    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zmrpl" event={"ID":"d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d","Type":"ContainerStarted","Data":"10e2a66afec73116005bf39c2b4c2e6891865c898a386b4ebf1ae3d44d9b04ba"}
May 25 23:36:14.353907 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:14.353852    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:36:14.355578 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:14.355551    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8" event={"ID":"63f26bc5-73a4-4b71-833a-c009538f5068","Type":"ContainerStarted","Data":"4887390f309e66e8a5e8591a5c37657eb57b3cd55f438da5f1ac3b3ede94f2b4"}
May 25 23:36:14.355675 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:14.355589    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8" event={"ID":"63f26bc5-73a4-4b71-833a-c009538f5068","Type":"ContainerStarted","Data":"119bc169093accbe3c660efb49ee9a7215131aa4f6d79c6227e00b4950c237c8"}
May 25 23:36:14.355675 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:14.355601    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8" event={"ID":"63f26bc5-73a4-4b71-833a-c009538f5068","Type":"ContainerStarted","Data":"ab2fcfce41ebe140f0eadf336a821bdb218c4d218c01d12976fe42a003ed0d2d"}
May 25 23:36:14.369433 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:14.369393    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9s245" podStartSLOduration=66.314312821 podStartE2EDuration="1m12.369379606s" podCreationTimestamp="2026-05-25 23:35:02 +0000 UTC" firstStartedPulling="2026-05-25 23:36:07.203803379 +0000 UTC m=+66.758747618" lastFinishedPulling="2026-05-25 23:36:13.258870147 +0000 UTC m=+72.813814403" observedRunningTime="2026-05-25 23:36:14.368647863 +0000 UTC m=+73.923592124" watchObservedRunningTime="2026-05-25 23:36:14.369379606 +0000 UTC m=+73.924323868"
May 25 23:36:14.386265 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:14.386221    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zmrpl" podStartSLOduration=67.561209439 podStartE2EDuration="1m13.386203993s" podCreationTimestamp="2026-05-25 23:35:01 +0000 UTC" firstStartedPulling="2026-05-25 23:36:07.440555139 +0000 UTC m=+66.995499387" lastFinishedPulling="2026-05-25 23:36:13.2655497 +0000 UTC m=+72.820493941" observedRunningTime="2026-05-25 23:36:14.385171217 +0000 UTC m=+73.940115478" watchObservedRunningTime="2026-05-25 23:36:14.386203993 +0000 UTC m=+73.941148253"
May 25 23:36:14.409881 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:14.409823    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6846f6469c-gd6v8" podStartSLOduration=2.643726986 podStartE2EDuration="7.40980597s" podCreationTimestamp="2026-05-25 23:36:07 +0000 UTC" firstStartedPulling="2026-05-25 23:36:08.498183715 +0000 UTC m=+68.053127967" lastFinishedPulling="2026-05-25 23:36:13.264262698 +0000 UTC m=+72.819206951" observedRunningTime="2026-05-25 23:36:14.407617437 +0000 UTC m=+73.962561737" watchObservedRunningTime="2026-05-25 23:36:14.40980597 +0000 UTC m=+73.964750231"
May 25 23:36:16.343133 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:16.343027    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57c58f7bb8-xvqmp"
May 25 23:36:26.897635 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:26.897596    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:26.897635 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:26.897645    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:45.361290 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:45.361263    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zmrpl"
May 25 23:36:46.902690 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:46.902659    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:36:46.906563 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:36:46.906538    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6dfc7747c8-kxcqg"
May 25 23:37:22.554621 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:22.554591    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 23:37:22.555291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:22.555118    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy-metric" containerID="cri-o://2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9" gracePeriod=120
May 25 23:37:22.555291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:22.555124    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy-web" containerID="cri-o://8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027" gracePeriod=120
May 25 23:37:22.555291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:22.555152    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy" containerID="cri-o://7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8" gracePeriod=120
May 25 23:37:22.555291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:22.555176    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="config-reloader" containerID="cri-o://25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8" gracePeriod=120
May 25 23:37:22.555291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:22.555096    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="alertmanager" containerID="cri-o://42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62" gracePeriod=120
May 25 23:37:22.555291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:22.555248    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="prom-label-proxy" containerID="cri-o://b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b" gracePeriod=120
May 25 23:37:23.557255 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:23.557218    2578 generic.go:358] "Generic (PLEG): container finished" podID="32db75f5-a85b-4778-91fd-599684318068" containerID="b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b" exitCode=0
May 25 23:37:23.557255 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:23.557246    2578 generic.go:358] "Generic (PLEG): container finished" podID="32db75f5-a85b-4778-91fd-599684318068" containerID="7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8" exitCode=0
May 25 23:37:23.557255 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:23.557252    2578 generic.go:358] "Generic (PLEG): container finished" podID="32db75f5-a85b-4778-91fd-599684318068" containerID="25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8" exitCode=0
May 25 23:37:23.557255 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:23.557258    2578 generic.go:358] "Generic (PLEG): container finished" podID="32db75f5-a85b-4778-91fd-599684318068" containerID="42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62" exitCode=0
May 25 23:37:23.557767 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:23.557294    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerDied","Data":"b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b"}
May 25 23:37:23.557767 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:23.557336    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerDied","Data":"7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8"}
May 25 23:37:23.557767 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:23.557352    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerDied","Data":"25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8"}
May 25 23:37:23.557767 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:23.557368    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerDied","Data":"42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62"}
May 25 23:37:23.904680 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:23.904653    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.068956 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.068916    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.068956 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.068954    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-cluster-tls-config\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069184 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.068982    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-alertmanager-trusted-ca-bundle\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069184 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069007    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-metric\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069184 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069025    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-config-out\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069184 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069047    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-web\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069184 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069071    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-config-volume\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069184 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069099    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-main-tls\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069184 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069135    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-metrics-client-ca\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069184 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069177    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cv5b\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-kube-api-access-6cv5b\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069592 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069206    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-alertmanager-main-db\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069592 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069231    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-tls-assets\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069592 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069259    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-web-config\") pod \"32db75f5-a85b-4778-91fd-599684318068\" (UID: \"32db75f5-a85b-4778-91fd-599684318068\") "
May 25 23:37:24.069592 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069491    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 23:37:24.070388 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.069895    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 23:37:24.070388 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.070097    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:37:24.072247 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.072048    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:37:24.072429 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.072366    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-config-out" (OuterVolumeSpecName: "config-out") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:37:24.072429 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.072392    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-config-volume" (OuterVolumeSpecName: "config-volume") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:37:24.072696 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.072481    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:37:24.072791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.072753    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-kube-api-access-6cv5b" (OuterVolumeSpecName: "kube-api-access-6cv5b") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "kube-api-access-6cv5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:37:24.072791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.072752    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:37:24.074360 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.074325    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:37:24.074509 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.074490    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:37:24.076378 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.076343    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:37:24.081670 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.081648    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-web-config" (OuterVolumeSpecName: "web-config") pod "32db75f5-a85b-4778-91fd-599684318068" (UID: "32db75f5-a85b-4778-91fd-599684318068"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:37:24.170297 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170251    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-alertmanager-main-db\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170297 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170290    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-tls-assets\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170297 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170305    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-web-config\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170320    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170334    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-cluster-tls-config\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170348    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170362    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170378    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32db75f5-a85b-4778-91fd-599684318068-config-out\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170393    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170408    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170423    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32db75f5-a85b-4778-91fd-599684318068-secret-alertmanager-main-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170437    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32db75f5-a85b-4778-91fd-599684318068-metrics-client-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.170549 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.170452    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cv5b\" (UniqueName: \"kubernetes.io/projected/32db75f5-a85b-4778-91fd-599684318068-kube-api-access-6cv5b\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:37:24.565398 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.565363    2578 generic.go:358] "Generic (PLEG): container finished" podID="32db75f5-a85b-4778-91fd-599684318068" containerID="2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9" exitCode=0
May 25 23:37:24.565398 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.565390    2578 generic.go:358] "Generic (PLEG): container finished" podID="32db75f5-a85b-4778-91fd-599684318068" containerID="8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027" exitCode=0
May 25 23:37:24.565898 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.565447    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerDied","Data":"2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9"}
May 25 23:37:24.565898 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.565557    2578 scope.go:117] "RemoveContainer" containerID="b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b"
May 25 23:37:24.565898 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.565661    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerDied","Data":"8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027"}
May 25 23:37:24.565898 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.565873    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32db75f5-a85b-4778-91fd-599684318068","Type":"ContainerDied","Data":"7ad5997e5e9893395c553252fe45d7c46c1765d00fd96b82f9c5ae2c169b73a3"}
May 25 23:37:24.566089 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.565979    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.577272 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.577249    2578 scope.go:117] "RemoveContainer" containerID="2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9"
May 25 23:37:24.584365 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.584344    2578 scope.go:117] "RemoveContainer" containerID="7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8"
May 25 23:37:24.590382 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.590357    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 23:37:24.591250 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.591221    2578 scope.go:117] "RemoveContainer" containerID="8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027"
May 25 23:37:24.595841 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.595818    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 23:37:24.598385 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.598368    2578 scope.go:117] "RemoveContainer" containerID="25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8"
May 25 23:37:24.604844 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.604828    2578 scope.go:117] "RemoveContainer" containerID="42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62"
May 25 23:37:24.611119 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.611098    2578 scope.go:117] "RemoveContainer" containerID="e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c"
May 25 23:37:24.618037 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.618014    2578 scope.go:117] "RemoveContainer" containerID="b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b"
May 25 23:37:24.618307 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:37:24.618286    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b\": container with ID starting with b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b not found: ID does not exist" containerID="b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b"
May 25 23:37:24.618356 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.618317    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b"} err="failed to get container status \"b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b\": rpc error: code = NotFound desc = could not find container \"b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b\": container with ID starting with b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b not found: ID does not exist"
May 25 23:37:24.618356 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.618353    2578 scope.go:117] "RemoveContainer" containerID="2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9"
May 25 23:37:24.618577 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:37:24.618561    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9\": container with ID starting with 2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9 not found: ID does not exist" containerID="2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9"
May 25 23:37:24.618633 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.618578    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9"} err="failed to get container status \"2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9\": rpc error: code = NotFound desc = could not find container \"2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9\": container with ID starting with 2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9 not found: ID does not exist"
May 25 23:37:24.618633 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.618589    2578 scope.go:117] "RemoveContainer" containerID="7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8"
May 25 23:37:24.618831 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:37:24.618816    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8\": container with ID starting with 7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8 not found: ID does not exist" containerID="7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8"
May 25 23:37:24.618878 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.618835    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8"} err="failed to get container status \"7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8\": rpc error: code = NotFound desc = could not find container \"7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8\": container with ID starting with 7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8 not found: ID does not exist"
May 25 23:37:24.618878 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.618848    2578 scope.go:117] "RemoveContainer" containerID="8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027"
May 25 23:37:24.619047 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:37:24.619030    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027\": container with ID starting with 8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027 not found: ID does not exist" containerID="8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027"
May 25 23:37:24.619089 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.619050    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027"} err="failed to get container status \"8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027\": rpc error: code = NotFound desc = could not find container \"8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027\": container with ID starting with 8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027 not found: ID does not exist"
May 25 23:37:24.619089 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.619063    2578 scope.go:117] "RemoveContainer" containerID="25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8"
May 25 23:37:24.619251 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:37:24.619233    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8\": container with ID starting with 25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8 not found: ID does not exist" containerID="25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8"
May 25 23:37:24.619291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.619254    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8"} err="failed to get container status \"25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8\": rpc error: code = NotFound desc = could not find container \"25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8\": container with ID starting with 25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8 not found: ID does not exist"
May 25 23:37:24.619291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.619265    2578 scope.go:117] "RemoveContainer" containerID="42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62"
May 25 23:37:24.619498 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:37:24.619475    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62\": container with ID starting with 42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62 not found: ID does not exist" containerID="42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62"
May 25 23:37:24.619590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.619507    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62"} err="failed to get container status \"42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62\": rpc error: code = NotFound desc = could not find container \"42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62\": container with ID starting with 42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62 not found: ID does not exist"
May 25 23:37:24.619590 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.619530    2578 scope.go:117] "RemoveContainer" containerID="e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c"
May 25 23:37:24.619826 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:37:24.619793    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c\": container with ID starting with e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c not found: ID does not exist" containerID="e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c"
May 25 23:37:24.619900 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.619833    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c"} err="failed to get container status \"e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c\": rpc error: code = NotFound desc = could not find container \"e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c\": container with ID starting with e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c not found: ID does not exist"
May 25 23:37:24.619900 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.619854    2578 scope.go:117] "RemoveContainer" containerID="b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b"
May 25 23:37:24.620256 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.620220    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b"} err="failed to get container status \"b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b\": rpc error: code = NotFound desc = could not find container \"b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b\": container with ID starting with b3f78420f86ab6c1cd99140c29ffe25013962198401a9ad9448b16dce2bd5d1b not found: ID does not exist"
May 25 23:37:24.620256 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.620244    2578 scope.go:117] "RemoveContainer" containerID="2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9"
May 25 23:37:24.620511 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.620489    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9"} err="failed to get container status \"2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9\": rpc error: code = NotFound desc = could not find container \"2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9\": container with ID starting with 2a89361d9319f15e67174cfb11984eb172e64af7fedf9752f32c23f60f2d02e9 not found: ID does not exist"
May 25 23:37:24.620611 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.620592    2578 scope.go:117] "RemoveContainer" containerID="7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8"
May 25 23:37:24.620940 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.620906    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8"} err="failed to get container status \"7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8\": rpc error: code = NotFound desc = could not find container \"7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8\": container with ID starting with 7c3995be3e86b109114f69c996a701a8cb28095bc650819aed2798ae73f4fcf8 not found: ID does not exist"
May 25 23:37:24.620940 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.620931    2578 scope.go:117] "RemoveContainer" containerID="8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027"
May 25 23:37:24.621185 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621163    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027"} err="failed to get container status \"8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027\": rpc error: code = NotFound desc = could not find container \"8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027\": container with ID starting with 8a995ee13b96dcf6849a29ea8b65f7a2399e900a957e7cc45b5c1b1cde4c9027 not found: ID does not exist"
May 25 23:37:24.621257 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621187    2578 scope.go:117] "RemoveContainer" containerID="25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8"
May 25 23:37:24.621367 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621351    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 23:37:24.621467 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621448    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8"} err="failed to get container status \"25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8\": rpc error: code = NotFound desc = could not find container \"25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8\": container with ID starting with 25e7467d710abf9becb5dedc2c8d80c99d5d1adbc5a870f1d09b04bab1259cb8 not found: ID does not exist"
May 25 23:37:24.621510 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621468    2578 scope.go:117] "RemoveContainer" containerID="42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62"
May 25 23:37:24.621689 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621673    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="alertmanager"
May 25 23:37:24.621772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621692    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="alertmanager"
May 25 23:37:24.621772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621685    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62"} err="failed to get container status \"42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62\": rpc error: code = NotFound desc = could not find container \"42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62\": container with ID starting with 42691f40983b3dba8689f470391d163808849c6574154995c5494c94bd909e62 not found: ID does not exist"
May 25 23:37:24.621772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621707    2578 scope.go:117] "RemoveContainer" containerID="e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c"
May 25 23:37:24.621772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621714    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy-web"
May 25 23:37:24.621772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621740    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy-web"
May 25 23:37:24.621772 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621768    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621776    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621789    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="init-config-reloader"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621797    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="init-config-reloader"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621806    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="prom-label-proxy"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621813    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="prom-label-proxy"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621823    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="config-reloader"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621830    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="config-reloader"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621839    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy-metric"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621847    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy-metric"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621918    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy-web"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621930    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="prom-label-proxy"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621942    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621950    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="alertmanager"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621959    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="config-reloader"
May 25 23:37:24.622006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.621967    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="32db75f5-a85b-4778-91fd-599684318068" containerName="kube-rbac-proxy-metric"
May 25 23:37:24.622535 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.622108    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c"} err="failed to get container status \"e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c\": rpc error: code = NotFound desc = could not find container \"e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c\": container with ID starting with e155fb68f0dc05a400a7d49085776bf783473cad3155500e51cc704bd0d90e5c not found: ID does not exist"
May 25 23:37:24.625826 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.625809    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.628667 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.628644    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 23:37:24.628805 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.628693    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8hnsp\""
May 25 23:37:24.628805 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.628702    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 23:37:24.628805 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.628713    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 23:37:24.628805 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.628755    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 23:37:24.629025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.629009    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 23:37:24.629089 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.629012    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 23:37:24.629150 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.629012    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 23:37:24.629410 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.629397    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 23:37:24.635854 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.635837    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 23:37:24.640835 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.640814    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 23:37:24.775190 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/563d4d67-c7aa-43cc-b6f2-744f2607c29b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775190 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-config-volume\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-web-config\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/563d4d67-c7aa-43cc-b6f2-744f2607c29b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/563d4d67-c7aa-43cc-b6f2-744f2607c29b-config-out\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775409 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/563d4d67-c7aa-43cc-b6f2-744f2607c29b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775670 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/563d4d67-c7aa-43cc-b6f2-744f2607c29b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.775670 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.775461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhqw\" (UniqueName: \"kubernetes.io/projected/563d4d67-c7aa-43cc-b6f2-744f2607c29b-kube-api-access-5hhqw\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.876776 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.876651    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/563d4d67-c7aa-43cc-b6f2-744f2607c29b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.876776 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.876688    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877018 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.876782    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877018 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.876833    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/563d4d67-c7aa-43cc-b6f2-744f2607c29b-config-out\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877018 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.876866    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877018 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.876896    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877232 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.877016    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/563d4d67-c7aa-43cc-b6f2-744f2607c29b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877232 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.877079    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/563d4d67-c7aa-43cc-b6f2-744f2607c29b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877232 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.877108    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhqw\" (UniqueName: \"kubernetes.io/projected/563d4d67-c7aa-43cc-b6f2-744f2607c29b-kube-api-access-5hhqw\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877232 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.877157    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/563d4d67-c7aa-43cc-b6f2-744f2607c29b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877232 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.877191    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877232 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.877225    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-config-volume\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.877534 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.877261    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-web-config\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.878337 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.878040    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/563d4d67-c7aa-43cc-b6f2-744f2607c29b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.879037 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.878489    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/563d4d67-c7aa-43cc-b6f2-744f2607c29b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.879037 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.878994    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/563d4d67-c7aa-43cc-b6f2-744f2607c29b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.879670 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.879643    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/563d4d67-c7aa-43cc-b6f2-744f2607c29b-config-out\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.879941 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.879918    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.880099 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.880070    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.880214 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.880103    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/563d4d67-c7aa-43cc-b6f2-744f2607c29b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.880280 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.880217    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.880467 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.880448    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.880983 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.880965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-web-config\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.881464 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.881440    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-config-volume\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.881576 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.881559    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/563d4d67-c7aa-43cc-b6f2-744f2607c29b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.886506 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.886483    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhqw\" (UniqueName: \"kubernetes.io/projected/563d4d67-c7aa-43cc-b6f2-744f2607c29b-kube-api-access-5hhqw\") pod \"alertmanager-main-0\" (UID: \"563d4d67-c7aa-43cc-b6f2-744f2607c29b\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:24.935555 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:24.935527    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 23:37:25.007539 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:25.007507    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32db75f5-a85b-4778-91fd-599684318068" path="/var/lib/kubelet/pods/32db75f5-a85b-4778-91fd-599684318068/volumes"
May 25 23:37:25.072789 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:25.072765    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 23:37:25.074589 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:37:25.074562    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563d4d67_c7aa_43cc_b6f2_744f2607c29b.slice/crio-4accdf4c67a7b5f6fb063b4c9962c147b236b33e18c6e530c13cf0efaf433b36 WatchSource:0}: Error finding container 4accdf4c67a7b5f6fb063b4c9962c147b236b33e18c6e530c13cf0efaf433b36: Status 404 returned error can't find the container with id 4accdf4c67a7b5f6fb063b4c9962c147b236b33e18c6e530c13cf0efaf433b36
May 25 23:37:25.570792 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:25.570758    2578 generic.go:358] "Generic (PLEG): container finished" podID="563d4d67-c7aa-43cc-b6f2-744f2607c29b" containerID="e95206bb262a7dfbc399962984ab9b50d796330b6a3ae325c6772321ab5a1d57" exitCode=0
May 25 23:37:25.570792 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:25.570795    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563d4d67-c7aa-43cc-b6f2-744f2607c29b","Type":"ContainerDied","Data":"e95206bb262a7dfbc399962984ab9b50d796330b6a3ae325c6772321ab5a1d57"}
May 25 23:37:25.571223 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:25.570818    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563d4d67-c7aa-43cc-b6f2-744f2607c29b","Type":"ContainerStarted","Data":"4accdf4c67a7b5f6fb063b4c9962c147b236b33e18c6e530c13cf0efaf433b36"}
May 25 23:37:26.577120 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:26.577085    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563d4d67-c7aa-43cc-b6f2-744f2607c29b","Type":"ContainerStarted","Data":"0739127c94b40e5c005ae737ca3babd10834ed7f1d01c2af07758c55735c4e48"}
May 25 23:37:26.577120 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:26.577124    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563d4d67-c7aa-43cc-b6f2-744f2607c29b","Type":"ContainerStarted","Data":"1a712cb38c0d7df7e98507af143c4f72f0aa941e440b524f2006e282bbb53f30"}
May 25 23:37:26.577521 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:26.577136    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563d4d67-c7aa-43cc-b6f2-744f2607c29b","Type":"ContainerStarted","Data":"2f3309aea68032c0ae9b0cdb3ab26259d38014369c7ecb6b4d094412a4b44983"}
May 25 23:37:26.577521 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:26.577144    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563d4d67-c7aa-43cc-b6f2-744f2607c29b","Type":"ContainerStarted","Data":"32c24b19e547e21fa59b960b7ad6f95e7ecb290aee7328e178241395e11c52f6"}
May 25 23:37:26.577521 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:26.577152    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563d4d67-c7aa-43cc-b6f2-744f2607c29b","Type":"ContainerStarted","Data":"5eb97deb94b835c92e4d91dd0b81838a3ef0e5d4990f2b2e92ddfaf63df91fc7"}
May 25 23:37:26.577521 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:26.577160    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563d4d67-c7aa-43cc-b6f2-744f2607c29b","Type":"ContainerStarted","Data":"966c37786ad5ad46b4000d41585bd58ab1ce2985b13af93df6951acac05348fa"}
May 25 23:37:26.606034 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:37:26.605984    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.605970176 podStartE2EDuration="2.605970176s" podCreationTimestamp="2026-05-25 23:37:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:37:26.604092062 +0000 UTC m=+146.159036324" watchObservedRunningTime="2026-05-25 23:37:26.605970176 +0000 UTC m=+146.160914446"
May 25 23:40:00.922480 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:40:00.922450    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:40:00.923017 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:40:00.922450    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:40:00.925974 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:40:00.925956    2578 kubelet.go:1628] "Image garbage collection succeeded"
May 25 23:41:27.361040 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.360998    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"]
May 25 23:41:27.364291 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.364270    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"
May 25 23:41:27.366863 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.366846    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 25 23:41:27.367762 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.367744    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-vbx79\""
May 25 23:41:27.367762 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.367756    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 25 23:41:27.373826 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.373802    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"]
May 25 23:41:27.488803 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.488769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg4l5\" (UniqueName: \"kubernetes.io/projected/80473cc5-d469-4fdb-a1a4-8e04be8e7c8a-kube-api-access-fg4l5\") pod \"feast-operator-controller-manager-6bfd8475f9-z7vt7\" (UID: \"80473cc5-d469-4fdb-a1a4-8e04be8e7c8a\") " pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"
May 25 23:41:27.590054 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.590028    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fg4l5\" (UniqueName: \"kubernetes.io/projected/80473cc5-d469-4fdb-a1a4-8e04be8e7c8a-kube-api-access-fg4l5\") pod \"feast-operator-controller-manager-6bfd8475f9-z7vt7\" (UID: \"80473cc5-d469-4fdb-a1a4-8e04be8e7c8a\") " pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"
May 25 23:41:27.599308 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.599281    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg4l5\" (UniqueName: \"kubernetes.io/projected/80473cc5-d469-4fdb-a1a4-8e04be8e7c8a-kube-api-access-fg4l5\") pod \"feast-operator-controller-manager-6bfd8475f9-z7vt7\" (UID: \"80473cc5-d469-4fdb-a1a4-8e04be8e7c8a\") " pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"
May 25 23:41:27.674957 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.674873    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"
May 25 23:41:27.800491 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.800461    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"]
May 25 23:41:27.804868 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:41:27.804832    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80473cc5_d469_4fdb_a1a4_8e04be8e7c8a.slice/crio-4c4e3dcafabe549e5cb6bd8f2b14fa77ab6d26f631fe4ffdd1702e21f232cbbf WatchSource:0}: Error finding container 4c4e3dcafabe549e5cb6bd8f2b14fa77ab6d26f631fe4ffdd1702e21f232cbbf: Status 404 returned error can't find the container with id 4c4e3dcafabe549e5cb6bd8f2b14fa77ab6d26f631fe4ffdd1702e21f232cbbf
May 25 23:41:27.806552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:27.806531    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 23:41:28.241193 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:28.241150    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7" event={"ID":"80473cc5-d469-4fdb-a1a4-8e04be8e7c8a","Type":"ContainerStarted","Data":"4c4e3dcafabe549e5cb6bd8f2b14fa77ab6d26f631fe4ffdd1702e21f232cbbf"}
May 25 23:41:31.254305 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:31.254255    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7" event={"ID":"80473cc5-d469-4fdb-a1a4-8e04be8e7c8a","Type":"ContainerStarted","Data":"a982ac2d2965ec33397e45b50124e1f38618f4a9b14fc8ba277651feb0db1f07"}
May 25 23:41:31.254750 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:31.254366    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"
May 25 23:41:31.271336 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:31.271278    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7" podStartSLOduration=1.84269988 podStartE2EDuration="4.271262596s" podCreationTimestamp="2026-05-25 23:41:27 +0000 UTC" firstStartedPulling="2026-05-25 23:41:27.806662281 +0000 UTC m=+387.361606520" lastFinishedPulling="2026-05-25 23:41:30.235224997 +0000 UTC m=+389.790169236" observedRunningTime="2026-05-25 23:41:31.270199341 +0000 UTC m=+390.825143601" watchObservedRunningTime="2026-05-25 23:41:31.271262596 +0000 UTC m=+390.826206856"
May 25 23:41:42.259480 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:41:42.259450    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"
May 25 23:42:10.072741 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.072689    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"]
May 25 23:42:10.077778 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.077754    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.080509 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.080486    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 25 23:42:10.081518 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.081472    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 25 23:42:10.081518 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.081504    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 25 23:42:10.081698 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.081609    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 25 23:42:10.081890 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.081875    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 25 23:42:10.081970 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.081934    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-rg67g\""
May 25 23:42:10.082471 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.082454    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 25 23:42:10.082471 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.082461    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 25 23:42:10.089037 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.089013    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"]
May 25 23:42:10.132558 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.132527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-ui-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.132741 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.132567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e039c9-e5d5-4afe-9b1e-d17f99518161-feast-data\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.132741 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.132653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-online-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.132741 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.132680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-registry-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.132741 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.132700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxb6f\" (UniqueName: \"kubernetes.io/projected/f5e039c9-e5d5-4afe-9b1e-d17f99518161-kube-api-access-rxb6f\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.132741 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.132734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-offline-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.233161 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.233128    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-ui-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.233161 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.233172    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e039c9-e5d5-4afe-9b1e-d17f99518161-feast-data\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.233495 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.233224    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-online-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.233495 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.233244    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-registry-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.233495 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.233274    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rxb6f\" (UniqueName: \"kubernetes.io/projected/f5e039c9-e5d5-4afe-9b1e-d17f99518161-kube-api-access-rxb6f\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.233495 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.233303    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-offline-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.233495 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:42:10.233370    2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 25 23:42:10.233495 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:42:10.233410    2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 25 23:42:10.233495 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:42:10.233370    2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 25 23:42:10.233852 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:42:10.233466    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-offline-tls podName:f5e039c9-e5d5-4afe-9b1e-d17f99518161 nodeName:}" failed. No retries permitted until 2026-05-25 23:42:10.733446162 +0000 UTC m=+430.288390420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-offline-tls") pod "feast-simple-feast-setup-665cb569fb-m8fcl" (UID: "f5e039c9-e5d5-4afe-9b1e-d17f99518161") : secret "feast-simple-feast-setup-offline-tls" not found
May 25 23:42:10.233852 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:42:10.233560    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-online-tls podName:f5e039c9-e5d5-4afe-9b1e-d17f99518161 nodeName:}" failed. No retries permitted until 2026-05-25 23:42:10.733540808 +0000 UTC m=+430.288485047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-online-tls") pod "feast-simple-feast-setup-665cb569fb-m8fcl" (UID: "f5e039c9-e5d5-4afe-9b1e-d17f99518161") : secret "feast-simple-feast-setup-online-tls" not found
May 25 23:42:10.233852 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:42:10.233576    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-registry-tls podName:f5e039c9-e5d5-4afe-9b1e-d17f99518161 nodeName:}" failed. No retries permitted until 2026-05-25 23:42:10.733567267 +0000 UTC m=+430.288511505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-registry-tls") pod "feast-simple-feast-setup-665cb569fb-m8fcl" (UID: "f5e039c9-e5d5-4afe-9b1e-d17f99518161") : secret "feast-simple-feast-setup-registry-tls" not found
May 25 23:42:10.233852 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.233685    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e039c9-e5d5-4afe-9b1e-d17f99518161-feast-data\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.235674 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.235655    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-ui-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.244155 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.244131    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxb6f\" (UniqueName: \"kubernetes.io/projected/f5e039c9-e5d5-4afe-9b1e-d17f99518161-kube-api-access-rxb6f\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.738407 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.738375    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-online-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.738407 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.738412    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-registry-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.738671 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.738433    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-offline-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.741011 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.740974    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-registry-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.741011 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.741002    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-online-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:10.741154 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:10.741032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-offline-tls\") pod \"feast-simple-feast-setup-665cb569fb-m8fcl\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:11.001594 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:11.001500    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:11.145512 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:11.145487    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"]
May 25 23:42:11.146585 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:42:11.146561    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5e039c9_e5d5_4afe_9b1e_d17f99518161.slice/crio-a7656b8f8263598551c01e07378988b298b117c3d46fd1563f8cb1aa411e2d8a WatchSource:0}: Error finding container a7656b8f8263598551c01e07378988b298b117c3d46fd1563f8cb1aa411e2d8a: Status 404 returned error can't find the container with id a7656b8f8263598551c01e07378988b298b117c3d46fd1563f8cb1aa411e2d8a
May 25 23:42:11.366059 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:11.366022    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerStarted","Data":"a7656b8f8263598551c01e07378988b298b117c3d46fd1563f8cb1aa411e2d8a"}
May 25 23:42:39.454906 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:39.454858    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerStarted","Data":"5040b55ca8976df527bbe4c981ac3798dbfdd767e4ff5737a3a5598a94f69901"}
May 25 23:42:44.470577 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:44.470542    2578 generic.go:358] "Generic (PLEG): container finished" podID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerID="5040b55ca8976df527bbe4c981ac3798dbfdd767e4ff5737a3a5598a94f69901" exitCode=0
May 25 23:42:44.471086 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:44.470615    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerDied","Data":"5040b55ca8976df527bbe4c981ac3798dbfdd767e4ff5737a3a5598a94f69901"}
May 25 23:42:45.476099 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:45.476061    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerStarted","Data":"dd3efc1714cdf98b0cc2df5245125c5217677abf56d888fb68fec5566a77b229"}
May 25 23:42:50.494614 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:50.494530    2578 generic.go:358] "Generic (PLEG): container finished" podID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerID="dd3efc1714cdf98b0cc2df5245125c5217677abf56d888fb68fec5566a77b229" exitCode=0
May 25 23:42:50.494614 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:50.494602    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerDied","Data":"dd3efc1714cdf98b0cc2df5245125c5217677abf56d888fb68fec5566a77b229"}
May 25 23:42:51.502438 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:51.502399    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerStarted","Data":"7e5bf5397babc6d96bd5b5774e0b9f6501f19d0cb585c9b3d660cfe1f20c5397"}
May 25 23:42:51.502955 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:51.502447    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerStarted","Data":"3e95d8c6ec929fd90ab28dd1bdc6d0bb357aa99ce7bef6c4928363d59a4941a3"}
May 25 23:42:51.502955 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:51.502462    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerStarted","Data":"60f4e3e0d31515c1e02a949609853de9afec4d51c79883c3bbc7d8cb032f8533"}
May 25 23:42:51.502955 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:51.502477    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerStarted","Data":"e358210ece3ac6e0f29f9ba3f9eda4be3207dbf2887b33df1dcb842593ca7b9c"}
May 25 23:42:51.529237 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:51.529170    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podStartSLOduration=13.587340312 podStartE2EDuration="41.529147113s" podCreationTimestamp="2026-05-25 23:42:10 +0000 UTC" firstStartedPulling="2026-05-25 23:42:11.148297493 +0000 UTC m=+430.703241732" lastFinishedPulling="2026-05-25 23:42:39.090104291 +0000 UTC m=+458.645048533" observedRunningTime="2026-05-25 23:42:51.526778701 +0000 UTC m=+471.081722965" watchObservedRunningTime="2026-05-25 23:42:51.529147113 +0000 UTC m=+471.084091375"
May 25 23:42:53.001991 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:53.001950    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:53.002476 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:53.002006    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:53.002476 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:53.002021    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:53.002476 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:53.002034    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:53.004437 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:53.004026    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.17:8816: connect: connection refused"
May 25 23:42:53.004437 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:53.004370    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.17:8443: connect: connection refused"
May 25 23:42:53.004437 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:53.004378    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.17:6571: connect: connection refused"
May 25 23:42:53.004871 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:53.004843    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="online" probeResult="failure" output="Get \"https://10.133.0.17:6567/health\": dial tcp 10.133.0.17:6567: connect: connection refused"
May 25 23:42:56.002413 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:56.002341    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.17:6571: connect: connection refused"
May 25 23:42:56.002933 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:56.002341    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.17:8443: connect: connection refused"
May 25 23:42:56.002933 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:56.002341    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.17:8816: connect: connection refused"
May 25 23:42:56.002933 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:56.002362    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="online" probeResult="failure" output="Get \"https://10.133.0.17:6567/health\": dial tcp 10.133.0.17:6567: connect: connection refused"
May 25 23:42:59.003146 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.003099    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.007963 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.007937    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.008123 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.007970    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.008123 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.007982    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.008123 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.007994    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.008123 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.008005    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.008123 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.008059    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.008123 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.008100    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.008928 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.008906    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.532031 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.531999    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.532343 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.532321    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:42:59.535031 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:42:59.535011    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:44:01.385829 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.385718    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"]
May 25 23:44:01.386322 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.386220    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="registry" containerID="cri-o://e358210ece3ac6e0f29f9ba3f9eda4be3207dbf2887b33df1dcb842593ca7b9c" gracePeriod=30
May 25 23:44:01.386322 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.386268    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="offline" containerID="cri-o://3e95d8c6ec929fd90ab28dd1bdc6d0bb357aa99ce7bef6c4928363d59a4941a3" gracePeriod=30
May 25 23:44:01.386463 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.386278    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="online" containerID="cri-o://60f4e3e0d31515c1e02a949609853de9afec4d51c79883c3bbc7d8cb032f8533" gracePeriod=30
May 25 23:44:01.386463 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.386389    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="ui" containerID="cri-o://7e5bf5397babc6d96bd5b5774e0b9f6501f19d0cb585c9b3d660cfe1f20c5397" gracePeriod=30
May 25 23:44:01.635340 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.635295    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"]
May 25 23:44:01.639507 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.639437    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.641941 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.641906    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-t9xts\""
May 25 23:44:01.653054 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.653014    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"]
May 25 23:44:01.691135 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.691100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-registry-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.691305 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.691159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-offline-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.691305 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.691235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-ui-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.691384 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.691319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-online-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.691384 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.691346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrb27\" (UniqueName: \"kubernetes.io/projected/34aeb034-bd54-4344-9bbf-436ea01161ec-kube-api-access-vrb27\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.691384 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.691379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/34aeb034-bd54-4344-9bbf-436ea01161ec-feast-data\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.792822 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.792783    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-online-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.792822 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.792826    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vrb27\" (UniqueName: \"kubernetes.io/projected/34aeb034-bd54-4344-9bbf-436ea01161ec-kube-api-access-vrb27\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.793071 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.792860    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/34aeb034-bd54-4344-9bbf-436ea01161ec-feast-data\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.793071 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.792917    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-registry-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.793071 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.792976    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-offline-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.793071 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.793004    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-ui-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.793264 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:44:01.792973    2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 25 23:44:01.793264 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:44:01.793162    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-online-tls podName:34aeb034-bd54-4344-9bbf-436ea01161ec nodeName:}" failed. No retries permitted until 2026-05-25 23:44:02.293138772 +0000 UTC m=+541.848083028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-online-tls") pod "feast-simple-feast-setup-665cb569fb-4j44j" (UID: "34aeb034-bd54-4344-9bbf-436ea01161ec") : secret "feast-simple-feast-setup-online-tls" not found
May 25 23:44:01.793264 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:44:01.793075    2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 25 23:44:01.793264 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:44:01.793252    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-registry-tls podName:34aeb034-bd54-4344-9bbf-436ea01161ec nodeName:}" failed. No retries permitted until 2026-05-25 23:44:02.293233111 +0000 UTC m=+541.848177369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-registry-tls") pod "feast-simple-feast-setup-665cb569fb-4j44j" (UID: "34aeb034-bd54-4344-9bbf-436ea01161ec") : secret "feast-simple-feast-setup-registry-tls" not found
May 25 23:44:01.793476 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.793376    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/34aeb034-bd54-4344-9bbf-436ea01161ec-feast-data\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.795632 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.795595    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-offline-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.795813 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.795778    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-ui-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:01.803384 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:01.803349    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrb27\" (UniqueName: \"kubernetes.io/projected/34aeb034-bd54-4344-9bbf-436ea01161ec-kube-api-access-vrb27\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:02.298707 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.298617    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-online-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:02.298707 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.298685    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-registry-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:02.301147 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.301119    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-online-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:02.301147 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.301140    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-registry-tls\") pod \"feast-simple-feast-setup-665cb569fb-4j44j\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") " pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:02.548786 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.548691    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:02.681797 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.681772    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"]
May 25 23:44:02.683276 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:44:02.683242    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34aeb034_bd54_4344_9bbf_436ea01161ec.slice/crio-7fce4b8cd396da8e03ad29143033ee6ffbdb99035f00ea961249aa1d76fa3849 WatchSource:0}: Error finding container 7fce4b8cd396da8e03ad29143033ee6ffbdb99035f00ea961249aa1d76fa3849: Status 404 returned error can't find the container with id 7fce4b8cd396da8e03ad29143033ee6ffbdb99035f00ea961249aa1d76fa3849
May 25 23:44:02.723075 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.723043    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerStarted","Data":"7fce4b8cd396da8e03ad29143033ee6ffbdb99035f00ea961249aa1d76fa3849"}
May 25 23:44:02.725350 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.725323    2578 generic.go:358] "Generic (PLEG): container finished" podID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerID="7e5bf5397babc6d96bd5b5774e0b9f6501f19d0cb585c9b3d660cfe1f20c5397" exitCode=0
May 25 23:44:02.725350 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.725345    2578 generic.go:358] "Generic (PLEG): container finished" podID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerID="60f4e3e0d31515c1e02a949609853de9afec4d51c79883c3bbc7d8cb032f8533" exitCode=0
May 25 23:44:02.725507 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.725376    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerDied","Data":"7e5bf5397babc6d96bd5b5774e0b9f6501f19d0cb585c9b3d660cfe1f20c5397"}
May 25 23:44:02.725507 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:02.725395    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerDied","Data":"60f4e3e0d31515c1e02a949609853de9afec4d51c79883c3bbc7d8cb032f8533"}
May 25 23:44:03.729522 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:03.729486    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerStarted","Data":"adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305"}
May 25 23:44:07.745989 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:07.745899    2578 generic.go:358] "Generic (PLEG): container finished" podID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerID="adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305" exitCode=0
May 25 23:44:07.745989 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:07.745975    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerDied","Data":"adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305"}
May 25 23:44:08.750395 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:08.750354    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerStarted","Data":"dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c"}
May 25 23:44:09.004468 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:09.004376    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.17:8443: connect: connection refused"
May 25 23:44:09.532987 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:09.532929    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="online" probeResult="failure" output="Get \"https://10.133.0.17:6567/health\": dial tcp 10.133.0.17:6567: connect: connection refused"
May 25 23:44:13.770665 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:13.770566    2578 generic.go:358] "Generic (PLEG): container finished" podID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerID="dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c" exitCode=0
May 25 23:44:13.771145 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:13.770651    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerDied","Data":"dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c"}
May 25 23:44:14.778621 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:14.778582    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerStarted","Data":"2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f"}
May 25 23:44:14.779143 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:14.778634    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerStarted","Data":"900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c"}
May 25 23:44:14.779143 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:14.778650    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerStarted","Data":"0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf"}
May 25 23:44:14.779143 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:14.778664    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerStarted","Data":"5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7"}
May 25 23:44:14.805084 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:14.805023    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podStartSLOduration=13.805002805 podStartE2EDuration="13.805002805s" podCreationTimestamp="2026-05-25 23:44:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:44:14.802318649 +0000 UTC m=+554.357262939" watchObservedRunningTime="2026-05-25 23:44:14.805002805 +0000 UTC m=+554.359947067"
May 25 23:44:17.549357 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:17.549314    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:17.549854 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:17.549379    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:17.549854 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:17.549396    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:17.549854 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:17.549408    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:17.551406 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:17.551367    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.18:6571: connect: connection refused"
May 25 23:44:17.551533 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:17.551402    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.18:8816: connect: connection refused"
May 25 23:44:17.551533 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:17.551503    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.18:8443: connect: connection refused"
May 25 23:44:17.551715 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:17.551672    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="online" probeResult="failure" output="Get \"https://10.133.0.18:6567/health\": dial tcp 10.133.0.18:6567: connect: connection refused"
May 25 23:44:19.004004 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:19.003958    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.17:8443: connect: connection refused"
May 25 23:44:19.532526 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:19.532468    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="online" probeResult="failure" output="Get \"https://10.133.0.17:6567/health\": dial tcp 10.133.0.17:6567: connect: connection refused"
May 25 23:44:20.549962 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.549925    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.550363 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.550069    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.550363 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.550315    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.550649 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.550622    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.550888 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.550834    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.550888 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.550853    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.551033 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.551013    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.551094 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.551081    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.551193 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.551179    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.554593 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.554574    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.800492 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.800403    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:20.803451 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:20.803428    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:44:29.004790 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:29.004752    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.17:8443: connect: connection refused"
May 25 23:44:29.007320 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:29.007298    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:44:29.532315 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:29.532270    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="online" probeResult="failure" output="Get \"https://10.133.0.17:6567/health\": dial tcp 10.133.0.17:6567: connect: connection refused"
May 25 23:44:29.532514 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:29.532412    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:44:31.835289 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:31.835259    2578 generic.go:358] "Generic (PLEG): container finished" podID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerID="3e95d8c6ec929fd90ab28dd1bdc6d0bb357aa99ce7bef6c4928363d59a4941a3" exitCode=137
May 25 23:44:31.835289 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:31.835287    2578 generic.go:358] "Generic (PLEG): container finished" podID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerID="e358210ece3ac6e0f29f9ba3f9eda4be3207dbf2887b33df1dcb842593ca7b9c" exitCode=137
May 25 23:44:31.835697 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:31.835334    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerDied","Data":"3e95d8c6ec929fd90ab28dd1bdc6d0bb357aa99ce7bef6c4928363d59a4941a3"}
May 25 23:44:31.835697 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:31.835371    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerDied","Data":"e358210ece3ac6e0f29f9ba3f9eda4be3207dbf2887b33df1dcb842593ca7b9c"}
May 25 23:44:32.036038 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.036017    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:44:32.176030 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.175938    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-offline-tls\") pod \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") "
May 25 23:44:32.176030 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.175977    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-registry-tls\") pod \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") "
May 25 23:44:32.176030 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.175999    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxb6f\" (UniqueName: \"kubernetes.io/projected/f5e039c9-e5d5-4afe-9b1e-d17f99518161-kube-api-access-rxb6f\") pod \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") "
May 25 23:44:32.176030 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.176033    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-online-tls\") pod \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") "
May 25 23:44:32.176339 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.176100    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e039c9-e5d5-4afe-9b1e-d17f99518161-feast-data\") pod \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") "
May 25 23:44:32.176339 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.176153    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-ui-tls\") pod \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\" (UID: \"f5e039c9-e5d5-4afe-9b1e-d17f99518161\") "
May 25 23:44:32.176782 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.176743    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e039c9-e5d5-4afe-9b1e-d17f99518161-feast-data" (OuterVolumeSpecName: "feast-data") pod "f5e039c9-e5d5-4afe-9b1e-d17f99518161" (UID: "f5e039c9-e5d5-4afe-9b1e-d17f99518161"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:44:32.178363 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.178331    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f5e039c9-e5d5-4afe-9b1e-d17f99518161" (UID: "f5e039c9-e5d5-4afe-9b1e-d17f99518161"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:44:32.178473 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.178386    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e039c9-e5d5-4afe-9b1e-d17f99518161-kube-api-access-rxb6f" (OuterVolumeSpecName: "kube-api-access-rxb6f") pod "f5e039c9-e5d5-4afe-9b1e-d17f99518161" (UID: "f5e039c9-e5d5-4afe-9b1e-d17f99518161"). InnerVolumeSpecName "kube-api-access-rxb6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:44:32.178473 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.178391    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f5e039c9-e5d5-4afe-9b1e-d17f99518161" (UID: "f5e039c9-e5d5-4afe-9b1e-d17f99518161"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:44:32.178578 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.178525    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f5e039c9-e5d5-4afe-9b1e-d17f99518161" (UID: "f5e039c9-e5d5-4afe-9b1e-d17f99518161"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:44:32.178615 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.178589    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-online-tls" (OuterVolumeSpecName: "online-tls") pod "f5e039c9-e5d5-4afe-9b1e-d17f99518161" (UID: "f5e039c9-e5d5-4afe-9b1e-d17f99518161"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:44:32.277813 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.277777    2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e039c9-e5d5-4afe-9b1e-d17f99518161-feast-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:44:32.277813 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.277808    2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-ui-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:44:32.277813 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.277817    2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-offline-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:44:32.277813 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.277825    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-registry-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:44:32.278065 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.277834    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxb6f\" (UniqueName: \"kubernetes.io/projected/f5e039c9-e5d5-4afe-9b1e-d17f99518161-kube-api-access-rxb6f\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:44:32.278065 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.277845    2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5e039c9-e5d5-4afe-9b1e-d17f99518161-online-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:44:32.840588 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.840555    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl" event={"ID":"f5e039c9-e5d5-4afe-9b1e-d17f99518161","Type":"ContainerDied","Data":"a7656b8f8263598551c01e07378988b298b117c3d46fd1563f8cb1aa411e2d8a"}
May 25 23:44:32.841128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.840604    2578 scope.go:117] "RemoveContainer" containerID="7e5bf5397babc6d96bd5b5774e0b9f6501f19d0cb585c9b3d660cfe1f20c5397"
May 25 23:44:32.841128 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.840694    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"
May 25 23:44:32.849137 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.849116    2578 scope.go:117] "RemoveContainer" containerID="3e95d8c6ec929fd90ab28dd1bdc6d0bb357aa99ce7bef6c4928363d59a4941a3"
May 25 23:44:32.856308 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.856292    2578 scope.go:117] "RemoveContainer" containerID="60f4e3e0d31515c1e02a949609853de9afec4d51c79883c3bbc7d8cb032f8533"
May 25 23:44:32.863855 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.863832    2578 scope.go:117] "RemoveContainer" containerID="e358210ece3ac6e0f29f9ba3f9eda4be3207dbf2887b33df1dcb842593ca7b9c"
May 25 23:44:32.864781 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.864759    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"]
May 25 23:44:32.868177 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.868155    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-m8fcl"]
May 25 23:44:32.874116 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.874093    2578 scope.go:117] "RemoveContainer" containerID="dd3efc1714cdf98b0cc2df5245125c5217677abf56d888fb68fec5566a77b229"
May 25 23:44:32.881283 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:32.881264    2578 scope.go:117] "RemoveContainer" containerID="5040b55ca8976df527bbe4c981ac3798dbfdd767e4ff5737a3a5598a94f69901"
May 25 23:44:33.007157 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:44:33.007120    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" path="/var/lib/kubelet/pods/f5e039c9-e5d5-4afe-9b1e-d17f99518161/volumes"
May 25 23:45:00.947801 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:00.947768    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:45:00.948950 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:00.948925    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:45:23.305958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.305869    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"]
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306184    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="online"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306197    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="online"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306206    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="registry"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306211    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="registry"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306219    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="feast-apply"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306227    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="feast-apply"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306235    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="offline"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306240    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="offline"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306260    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="feast-init"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306265    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="feast-init"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306270    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="ui"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306276    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="ui"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306332    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="registry"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306341    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="online"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306349    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="offline"
May 25 23:45:23.306397 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.306355    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5e039c9-e5d5-4afe-9b1e-d17f99518161" containerName="ui"
May 25 23:45:23.309769 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.309740    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.312344 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.312316    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 25 23:45:23.312503 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.312364    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 25 23:45:23.312503 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.312383    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 25 23:45:23.312503 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.312316    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 25 23:45:23.312503 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.312476    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-vbvzv\""
May 25 23:45:23.312700 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.312648    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 25 23:45:23.312700 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.312660    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 25 23:45:23.313229 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.313214    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 25 23:45:23.324817 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.324786    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"]
May 25 23:45:23.398037 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.397994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnv5p\" (UniqueName: \"kubernetes.io/projected/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-kube-api-access-bnv5p\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.398037 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.398041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-offline-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.398252 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.398068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-feast-data\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.398252 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.398100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-online-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.398252 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.398121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-ui-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.398252 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.398150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-registry-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.499462 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.499423    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-online-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.499462 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.499464    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-ui-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.499750 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.499509    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-registry-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.499750 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.499583    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bnv5p\" (UniqueName: \"kubernetes.io/projected/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-kube-api-access-bnv5p\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.499750 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.499612    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-offline-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.499750 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:45:23.499617    2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 25 23:45:23.499750 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.499646    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-feast-data\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.499750 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:45:23.499693    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-ui-tls podName:47c0a431-c16e-4392-a1d5-0ab4d49f7b90 nodeName:}" failed. No retries permitted until 2026-05-25 23:45:23.999671934 +0000 UTC m=+623.554616185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-ui-tls") pod "feast-simple-feast-remote-setup-6d9cff7794-mph9c" (UID: "47c0a431-c16e-4392-a1d5-0ab4d49f7b90") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 25 23:45:23.500103 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.500053    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-feast-data\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.500266 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.500249    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-registry-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.501984 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.501965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-offline-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.502045 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.502028    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-online-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:23.510490 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:23.510464    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnv5p\" (UniqueName: \"kubernetes.io/projected/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-kube-api-access-bnv5p\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:24.005016 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:24.004976    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-ui-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:24.007511 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:24.007483    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-ui-tls\") pod \"feast-simple-feast-remote-setup-6d9cff7794-mph9c\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:24.219450 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:24.219410    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:24.354761 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:24.354737    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"]
May 25 23:45:24.357056 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:45:24.357028    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47c0a431_c16e_4392_a1d5_0ab4d49f7b90.slice/crio-8d30c8f48c8f703d2aa6bc8f7ac4d0e8f2531283911fc805972c05eb089d6c69 WatchSource:0}: Error finding container 8d30c8f48c8f703d2aa6bc8f7ac4d0e8f2531283911fc805972c05eb089d6c69: Status 404 returned error can't find the container with id 8d30c8f48c8f703d2aa6bc8f7ac4d0e8f2531283911fc805972c05eb089d6c69
May 25 23:45:24.995456 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:24.995419    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerStarted","Data":"d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea"}
May 25 23:45:24.995456 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:24.995460    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerStarted","Data":"8d30c8f48c8f703d2aa6bc8f7ac4d0e8f2531283911fc805972c05eb089d6c69"}
May 25 23:45:30.014170 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:30.014137    2578 generic.go:358] "Generic (PLEG): container finished" podID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerID="d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea" exitCode=0
May 25 23:45:30.014593 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:30.014212    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerDied","Data":"d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea"}
May 25 23:45:31.019025 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:31.018987    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerStarted","Data":"f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3"}
May 25 23:45:36.035175 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:36.035144    2578 generic.go:358] "Generic (PLEG): container finished" podID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerID="f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3" exitCode=0
May 25 23:45:36.035602 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:36.035196    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerDied","Data":"f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3"}
May 25 23:45:37.041669 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:37.041631    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerStarted","Data":"56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53"}
May 25 23:45:37.041669 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:37.041676    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerStarted","Data":"090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf"}
May 25 23:45:37.042238 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:37.041686    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerStarted","Data":"1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5"}
May 25 23:45:37.064652 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:37.064570    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podStartSLOduration=14.064548363 podStartE2EDuration="14.064548363s" podCreationTimestamp="2026-05-25 23:45:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:45:37.063982405 +0000 UTC m=+636.618926669" watchObservedRunningTime="2026-05-25 23:45:37.064548363 +0000 UTC m=+636.619492625"
May 25 23:45:39.220584 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:39.220378    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:39.220584 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:39.220462    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:39.220584 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:39.220494    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:39.223222 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:39.222526    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.19:8816: connect: connection refused"
May 25 23:45:39.223222 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:39.222556    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.19:8443: connect: connection refused"
May 25 23:45:39.223222 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:39.222582    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="online" probeResult="failure" output="Get \"https://10.133.0.19:6567/health\": dial tcp 10.133.0.19:6567: connect: connection refused"
May 25 23:45:42.220966 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:42.220926    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:42.221480 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:42.221263    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:42.221693 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:42.221676    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:42.221794 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:42.221699    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:42.222114 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:42.222096    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:42.222179 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:42.222139    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:42.225426 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:42.225404    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:43.061876 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:43.061847    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:45:43.064951 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:45:43.064932    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:46:44.549750 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:44.549690    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"]
May 25 23:46:44.550609 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:44.550050    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="online" containerID="cri-o://1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5" gracePeriod=30
May 25 23:46:44.550609 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:44.550143    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="ui" containerID="cri-o://56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53" gracePeriod=30
May 25 23:46:44.550609 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:44.550194    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="offline" containerID="cri-o://090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf" gracePeriod=30
May 25 23:46:44.666450 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:44.666413    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"]
May 25 23:46:44.666945 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:44.666909    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="registry" containerID="cri-o://5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7" gracePeriod=30
May 25 23:46:44.667316 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:44.667221    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="offline" containerID="cri-o://900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c" gracePeriod=30
May 25 23:46:44.667591 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:44.667546    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="ui" containerID="cri-o://2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f" gracePeriod=30
May 25 23:46:44.667791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:44.667752    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="online" containerID="cri-o://0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf" gracePeriod=30
May 25 23:46:45.246790 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:45.246756    2578 generic.go:358] "Generic (PLEG): container finished" podID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerID="56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53" exitCode=0
May 25 23:46:45.246983 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:45.246833    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerDied","Data":"56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53"}
May 25 23:46:46.251907 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:46.251826    2578 generic.go:358] "Generic (PLEG): container finished" podID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerID="2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f" exitCode=0
May 25 23:46:46.251907 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:46.251852    2578 generic.go:358] "Generic (PLEG): container finished" podID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerID="0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf" exitCode=0
May 25 23:46:46.252358 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:46.251898    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerDied","Data":"2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f"}
May 25 23:46:46.252358 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:46.251937    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerDied","Data":"0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf"}
May 25 23:46:46.254196 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:46.254168    2578 generic.go:358] "Generic (PLEG): container finished" podID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerID="1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5" exitCode=0
May 25 23:46:46.254326 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:46.254206    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerDied","Data":"1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5"}
May 25 23:46:50.551213 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:50.551162    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.18:8443: connect: connection refused"
May 25 23:46:50.801717 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:50.801615    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="online" probeResult="failure" output="Get \"https://10.133.0.18:6567/health\": dial tcp 10.133.0.18:6567: connect: connection refused"
May 25 23:46:52.221906 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:52.221864    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.19:8443: connect: connection refused"
May 25 23:46:53.062026 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:46:53.061985    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="online" probeResult="failure" output="Get \"https://10.133.0.19:6567/health\": dial tcp 10.133.0.19:6567: connect: connection refused"
May 25 23:47:00.551114 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:00.551058    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.18:8443: connect: connection refused"
May 25 23:47:00.801609 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:00.801504    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="online" probeResult="failure" output="Get \"https://10.133.0.18:6567/health\": dial tcp 10.133.0.18:6567: connect: connection refused"
May 25 23:47:02.222173 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:02.222125    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.19:8443: connect: connection refused"
May 25 23:47:03.062820 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:03.062769    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="online" probeResult="failure" output="Get \"https://10.133.0.19:6567/health\": dial tcp 10.133.0.19:6567: connect: connection refused"
May 25 23:47:10.551072 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:10.551020    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.18:8443: connect: connection refused"
May 25 23:47:10.551561 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:10.551213    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:47:10.801538 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:10.801440    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="online" probeResult="failure" output="Get \"https://10.133.0.18:6567/health\": dial tcp 10.133.0.18:6567: connect: connection refused"
May 25 23:47:10.801690 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:10.801560    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:47:12.222626 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:12.222573    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.19:8443: connect: connection refused"
May 25 23:47:12.223085 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:12.222715    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:47:13.062231 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:13.062190    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="online" probeResult="failure" output="Get \"https://10.133.0.19:6567/health\": dial tcp 10.133.0.19:6567: connect: connection refused"
May 25 23:47:13.062411 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:13.062297    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:47:15.208598 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.208570    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:47:15.327268 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.327245    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:47:15.341101 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.341067    2578 generic.go:358] "Generic (PLEG): container finished" podID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerID="090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf" exitCode=137
May 25 23:47:15.341237 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.341137    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"
May 25 23:47:15.341237 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.341142    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerDied","Data":"090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf"}
May 25 23:47:15.341237 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.341179    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c" event={"ID":"47c0a431-c16e-4392-a1d5-0ab4d49f7b90","Type":"ContainerDied","Data":"8d30c8f48c8f703d2aa6bc8f7ac4d0e8f2531283911fc805972c05eb089d6c69"}
May 25 23:47:15.341237 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.341195    2578 scope.go:117] "RemoveContainer" containerID="56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53"
May 25 23:47:15.343783 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.343760    2578 generic.go:358] "Generic (PLEG): container finished" podID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerID="900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c" exitCode=137
May 25 23:47:15.343783 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.343780    2578 generic.go:358] "Generic (PLEG): container finished" podID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerID="5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7" exitCode=137
May 25 23:47:15.343950 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.343822    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"
May 25 23:47:15.343950 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.343844    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerDied","Data":"900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c"}
May 25 23:47:15.343950 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.343876    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerDied","Data":"5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7"}
May 25 23:47:15.343950 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.343889    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j" event={"ID":"34aeb034-bd54-4344-9bbf-436ea01161ec","Type":"ContainerDied","Data":"7fce4b8cd396da8e03ad29143033ee6ffbdb99035f00ea961249aa1d76fa3849"}
May 25 23:47:15.349588 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.349569    2578 scope.go:117] "RemoveContainer" containerID="090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf"
May 25 23:47:15.359101 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.359074    2578 scope.go:117] "RemoveContainer" containerID="1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5"
May 25 23:47:15.364452 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364419    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-online-tls\") pod \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") "
May 25 23:47:15.364562 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364466    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnv5p\" (UniqueName: \"kubernetes.io/projected/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-kube-api-access-bnv5p\") pod \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") "
May 25 23:47:15.364562 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364497    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/34aeb034-bd54-4344-9bbf-436ea01161ec-feast-data\") pod \"34aeb034-bd54-4344-9bbf-436ea01161ec\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") "
May 25 23:47:15.364636 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364596    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-feast-data\") pod \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") "
May 25 23:47:15.364636 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364622    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-registry-tls\") pod \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") "
May 25 23:47:15.364862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364657    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-offline-tls\") pod \"34aeb034-bd54-4344-9bbf-436ea01161ec\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") "
May 25 23:47:15.364862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364692    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-offline-tls\") pod \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") "
May 25 23:47:15.364862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364715    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-ui-tls\") pod \"34aeb034-bd54-4344-9bbf-436ea01161ec\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") "
May 25 23:47:15.364862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364795    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-online-tls\") pod \"34aeb034-bd54-4344-9bbf-436ea01161ec\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") "
May 25 23:47:15.364862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364823    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrb27\" (UniqueName: \"kubernetes.io/projected/34aeb034-bd54-4344-9bbf-436ea01161ec-kube-api-access-vrb27\") pod \"34aeb034-bd54-4344-9bbf-436ea01161ec\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") "
May 25 23:47:15.364862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364852    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-registry-tls\") pod \"34aeb034-bd54-4344-9bbf-436ea01161ec\" (UID: \"34aeb034-bd54-4344-9bbf-436ea01161ec\") "
May 25 23:47:15.365197 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364924    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-ui-tls\") pod \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\" (UID: \"47c0a431-c16e-4392-a1d5-0ab4d49f7b90\") "
May 25 23:47:15.365197 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.364982    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "47c0a431-c16e-4392-a1d5-0ab4d49f7b90" (UID: "47c0a431-c16e-4392-a1d5-0ab4d49f7b90"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 23:47:15.365197 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.365096    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aeb034-bd54-4344-9bbf-436ea01161ec-feast-data" (OuterVolumeSpecName: "feast-data") pod "34aeb034-bd54-4344-9bbf-436ea01161ec" (UID: "34aeb034-bd54-4344-9bbf-436ea01161ec"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:47:15.365354 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.365232    2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/34aeb034-bd54-4344-9bbf-436ea01161ec-feast-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.365354 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.365250    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-registry-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.365978 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.365949    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-feast-data" (OuterVolumeSpecName: "feast-data") pod "47c0a431-c16e-4392-a1d5-0ab4d49f7b90" (UID: "47c0a431-c16e-4392-a1d5-0ab4d49f7b90"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:47:15.367592 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.367553    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-kube-api-access-bnv5p" (OuterVolumeSpecName: "kube-api-access-bnv5p") pod "47c0a431-c16e-4392-a1d5-0ab4d49f7b90" (UID: "47c0a431-c16e-4392-a1d5-0ab4d49f7b90"). InnerVolumeSpecName "kube-api-access-bnv5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:47:15.368029 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.367999    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "34aeb034-bd54-4344-9bbf-436ea01161ec" (UID: "34aeb034-bd54-4344-9bbf-436ea01161ec"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:47:15.368552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.368467    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-online-tls" (OuterVolumeSpecName: "online-tls") pod "34aeb034-bd54-4344-9bbf-436ea01161ec" (UID: "34aeb034-bd54-4344-9bbf-436ea01161ec"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:47:15.368611 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.368556    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "47c0a431-c16e-4392-a1d5-0ab4d49f7b90" (UID: "47c0a431-c16e-4392-a1d5-0ab4d49f7b90"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:47:15.368611 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.368563    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "47c0a431-c16e-4392-a1d5-0ab4d49f7b90" (UID: "47c0a431-c16e-4392-a1d5-0ab4d49f7b90"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:47:15.368685 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.368635    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34aeb034-bd54-4344-9bbf-436ea01161ec-kube-api-access-vrb27" (OuterVolumeSpecName: "kube-api-access-vrb27") pod "34aeb034-bd54-4344-9bbf-436ea01161ec" (UID: "34aeb034-bd54-4344-9bbf-436ea01161ec"). InnerVolumeSpecName "kube-api-access-vrb27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:47:15.368747 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.368686    2578 scope.go:117] "RemoveContainer" containerID="f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3"
May 25 23:47:15.369473 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.369433    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "34aeb034-bd54-4344-9bbf-436ea01161ec" (UID: "34aeb034-bd54-4344-9bbf-436ea01161ec"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:47:15.369949 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.369916    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-online-tls" (OuterVolumeSpecName: "online-tls") pod "47c0a431-c16e-4392-a1d5-0ab4d49f7b90" (UID: "47c0a431-c16e-4392-a1d5-0ab4d49f7b90"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:47:15.370043 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.370028    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "34aeb034-bd54-4344-9bbf-436ea01161ec" (UID: "34aeb034-bd54-4344-9bbf-436ea01161ec"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:47:15.376692 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.376665    2578 scope.go:117] "RemoveContainer" containerID="d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea"
May 25 23:47:15.383874 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.383850    2578 scope.go:117] "RemoveContainer" containerID="56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53"
May 25 23:47:15.384139 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.384121    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53\": container with ID starting with 56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53 not found: ID does not exist" containerID="56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53"
May 25 23:47:15.384188 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.384155    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53"} err="failed to get container status \"56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53\": rpc error: code = NotFound desc = could not find container \"56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53\": container with ID starting with 56e630f31fdcebd78fc0259a86f5abd15a3ebcaaae19124094433d6daeba0c53 not found: ID does not exist"
May 25 23:47:15.384188 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.384177    2578 scope.go:117] "RemoveContainer" containerID="090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf"
May 25 23:47:15.384392 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.384371    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf\": container with ID starting with 090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf not found: ID does not exist" containerID="090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf"
May 25 23:47:15.384437 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.384397    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf"} err="failed to get container status \"090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf\": rpc error: code = NotFound desc = could not find container \"090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf\": container with ID starting with 090cc9826bcfeafaab3c2a92b8d69cecf428a8d31cf61e6fe0724ea71760e6cf not found: ID does not exist"
May 25 23:47:15.384437 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.384413    2578 scope.go:117] "RemoveContainer" containerID="1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5"
May 25 23:47:15.384596 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.384581    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5\": container with ID starting with 1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5 not found: ID does not exist" containerID="1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5"
May 25 23:47:15.384631 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.384599    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5"} err="failed to get container status \"1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5\": rpc error: code = NotFound desc = could not find container \"1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5\": container with ID starting with 1fb84babac2f142779e0a6e64811e001de364517cb699562229b2c6a988b13a5 not found: ID does not exist"
May 25 23:47:15.384631 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.384617    2578 scope.go:117] "RemoveContainer" containerID="f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3"
May 25 23:47:15.384935 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.384918    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3\": container with ID starting with f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3 not found: ID does not exist" containerID="f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3"
May 25 23:47:15.384991 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.384950    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3"} err="failed to get container status \"f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3\": rpc error: code = NotFound desc = could not find container \"f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3\": container with ID starting with f213157f12f619135bb0708bd3daaa0569c4a23d9f70022a59e715317d3cc1a3 not found: ID does not exist"
May 25 23:47:15.384991 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.384963    2578 scope.go:117] "RemoveContainer" containerID="d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea"
May 25 23:47:15.385154 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.385137    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea\": container with ID starting with d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea not found: ID does not exist" containerID="d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea"
May 25 23:47:15.385197 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.385157    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea"} err="failed to get container status \"d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea\": rpc error: code = NotFound desc = could not find container \"d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea\": container with ID starting with d2ec95c5c9d07d04f1256ee5f6d01a4048fd2cc99c91ef24fe43365c1ad460ea not found: ID does not exist"
May 25 23:47:15.385197 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.385171    2578 scope.go:117] "RemoveContainer" containerID="2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f"
May 25 23:47:15.392388 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.392372    2578 scope.go:117] "RemoveContainer" containerID="900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c"
May 25 23:47:15.399194 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.399173    2578 scope.go:117] "RemoveContainer" containerID="0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf"
May 25 23:47:15.405943 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.405912    2578 scope.go:117] "RemoveContainer" containerID="5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7"
May 25 23:47:15.412564 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.412536    2578 scope.go:117] "RemoveContainer" containerID="dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c"
May 25 23:47:15.419391 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.419363    2578 scope.go:117] "RemoveContainer" containerID="adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305"
May 25 23:47:15.426058 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.426039    2578 scope.go:117] "RemoveContainer" containerID="2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f"
May 25 23:47:15.426334 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.426314    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f\": container with ID starting with 2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f not found: ID does not exist" containerID="2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f"
May 25 23:47:15.426393 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.426342    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f"} err="failed to get container status \"2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f\": rpc error: code = NotFound desc = could not find container \"2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f\": container with ID starting with 2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f not found: ID does not exist"
May 25 23:47:15.426393 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.426362    2578 scope.go:117] "RemoveContainer" containerID="900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c"
May 25 23:47:15.426603 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.426587    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c\": container with ID starting with 900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c not found: ID does not exist" containerID="900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c"
May 25 23:47:15.426647 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.426609    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c"} err="failed to get container status \"900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c\": rpc error: code = NotFound desc = could not find container \"900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c\": container with ID starting with 900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c not found: ID does not exist"
May 25 23:47:15.426647 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.426628    2578 scope.go:117] "RemoveContainer" containerID="0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf"
May 25 23:47:15.426905 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.426887    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf\": container with ID starting with 0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf not found: ID does not exist" containerID="0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf"
May 25 23:47:15.426960 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.426910    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf"} err="failed to get container status \"0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf\": rpc error: code = NotFound desc = could not find container \"0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf\": container with ID starting with 0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf not found: ID does not exist"
May 25 23:47:15.426960 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.426924    2578 scope.go:117] "RemoveContainer" containerID="5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7"
May 25 23:47:15.427186 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.427164    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7\": container with ID starting with 5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7 not found: ID does not exist" containerID="5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7"
May 25 23:47:15.427230 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.427190    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7"} err="failed to get container status \"5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7\": rpc error: code = NotFound desc = could not find container \"5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7\": container with ID starting with 5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7 not found: ID does not exist"
May 25 23:47:15.427230 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.427204    2578 scope.go:117] "RemoveContainer" containerID="dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c"
May 25 23:47:15.427422 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.427406    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c\": container with ID starting with dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c not found: ID does not exist" containerID="dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c"
May 25 23:47:15.427467 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.427425    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c"} err="failed to get container status \"dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c\": rpc error: code = NotFound desc = could not find container \"dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c\": container with ID starting with dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c not found: ID does not exist"
May 25 23:47:15.427467 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.427440    2578 scope.go:117] "RemoveContainer" containerID="adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305"
May 25 23:47:15.427638 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:15.427618    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305\": container with ID starting with adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305 not found: ID does not exist" containerID="adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305"
May 25 23:47:15.427745 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.427643    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305"} err="failed to get container status \"adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305\": rpc error: code = NotFound desc = could not find container \"adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305\": container with ID starting with adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305 not found: ID does not exist"
May 25 23:47:15.427745 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.427658    2578 scope.go:117] "RemoveContainer" containerID="2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f"
May 25 23:47:15.427906 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.427884    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f"} err="failed to get container status \"2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f\": rpc error: code = NotFound desc = could not find container \"2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f\": container with ID starting with 2e81a0364e5c230ffee6e477bd9bafc9b494d89d3c1d4d6724125b0fab41f17f not found: ID does not exist"
May 25 23:47:15.427958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.427908    2578 scope.go:117] "RemoveContainer" containerID="900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c"
May 25 23:47:15.428138 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.428122    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c"} err="failed to get container status \"900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c\": rpc error: code = NotFound desc = could not find container \"900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c\": container with ID starting with 900eb82c5869e89fa147720f295418bad53ddb330128fb5b81b4cdd9f7895c6c not found: ID does not exist"
May 25 23:47:15.428138 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.428137    2578 scope.go:117] "RemoveContainer" containerID="0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf"
May 25 23:47:15.428348 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.428327    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf"} err="failed to get container status \"0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf\": rpc error: code = NotFound desc = could not find container \"0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf\": container with ID starting with 0c5aaaaa1ac4f7753ec4760fcd0a0aca07c408d4b39243563947f77d2dd873bf not found: ID does not exist"
May 25 23:47:15.428398 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.428348    2578 scope.go:117] "RemoveContainer" containerID="5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7"
May 25 23:47:15.428569 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.428553    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7"} err="failed to get container status \"5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7\": rpc error: code = NotFound desc = could not find container \"5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7\": container with ID starting with 5efde76ca05645ab95a1c9e3d301581074d3a716ac0f93d6cc5e2047884852e7 not found: ID does not exist"
May 25 23:47:15.428617 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.428569    2578 scope.go:117] "RemoveContainer" containerID="dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c"
May 25 23:47:15.428821 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.428802    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c"} err="failed to get container status \"dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c\": rpc error: code = NotFound desc = could not find container \"dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c\": container with ID starting with dd95e40c6459f36371569b11c0c341d16b33437475c60efe77692d6473c10b3c not found: ID does not exist"
May 25 23:47:15.428873 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.428822    2578 scope.go:117] "RemoveContainer" containerID="adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305"
May 25 23:47:15.429018 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.429000    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305"} err="failed to get container status \"adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305\": rpc error: code = NotFound desc = could not find container \"adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305\": container with ID starting with adb1f64ebd214f097e86ef5413be39fae600c7698d13b3340b035c079978d305 not found: ID does not exist"
May 25 23:47:15.465557 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465523    2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-feast-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.465557 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465554    2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-offline-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.465557 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465563    2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-offline-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.465791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465572    2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-ui-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.465791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465580    2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-online-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.465791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465590    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrb27\" (UniqueName: \"kubernetes.io/projected/34aeb034-bd54-4344-9bbf-436ea01161ec-kube-api-access-vrb27\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.465791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465599    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/34aeb034-bd54-4344-9bbf-436ea01161ec-registry-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.465791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465608    2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-ui-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.465791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465616    2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-online-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.465791 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.465624    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnv5p\" (UniqueName: \"kubernetes.io/projected/47c0a431-c16e-4392-a1d5-0ab4d49f7b90-kube-api-access-bnv5p\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:47:15.667171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.667137    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"]
May 25 23:47:15.670142 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.670105    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-665cb569fb-4j44j"]
May 25 23:47:15.680969 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.680943    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"]
May 25 23:47:15.687896 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:15.687865    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d9cff7794-mph9c"]
May 25 23:47:17.007499 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:17.007462    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" path="/var/lib/kubelet/pods/34aeb034-bd54-4344-9bbf-436ea01161ec/volumes"
May 25 23:47:17.008097 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:17.008083    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" path="/var/lib/kubelet/pods/47c0a431-c16e-4392-a1d5-0ab4d49f7b90/volumes"
May 25 23:47:27.732247 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732208    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-ns96t"]
May 25 23:47:27.732747 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732659    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="feast-init"
May 25 23:47:27.732747 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732676    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="feast-init"
May 25 23:47:27.732747 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732688    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="feast-apply"
May 25 23:47:27.732747 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732696    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="feast-apply"
May 25 23:47:27.732747 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732706    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="ui"
May 25 23:47:27.732747 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732715    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="ui"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732751    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="offline"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732759    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="offline"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732774    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="offline"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732782    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="offline"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732793    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="feast-apply"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732801    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="feast-apply"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732812    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="registry"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732821    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="registry"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732834    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="feast-init"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732842    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="feast-init"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732854    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="online"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732862    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="online"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732872    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="online"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732879    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="online"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732889    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="ui"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732898    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="ui"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732975    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="online"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732987    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="offline"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.732995    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="ui"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.733005    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="offline"
May 25 23:47:27.733012 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.733017    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="47c0a431-c16e-4392-a1d5-0ab4d49f7b90" containerName="online"
May 25 23:47:27.733675 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.733027    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="registry"
May 25 23:47:27.733675 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.733038    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="34aeb034-bd54-4344-9bbf-436ea01161ec" containerName="ui"
May 25 23:47:27.736301 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.736284    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:47:27.738633 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.738607    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 25 23:47:27.738633 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.738624    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-tgfmd\""
May 25 23:47:27.738862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.738648    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 25 23:47:27.738862 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.738651    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 25 23:47:27.742093 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.742070    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-ns96t"]
May 25 23:47:27.753529 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.753503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/16adc9ab-1dfd-47ae-9617-80fe07c16107-postgresdata\") pod \"postgres-7c7656cc6d-ns96t\" (UID: \"16adc9ab-1dfd-47ae-9617-80fe07c16107\") " pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:47:27.753529 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.753534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nxw5\" (UniqueName: \"kubernetes.io/projected/16adc9ab-1dfd-47ae-9617-80fe07c16107-kube-api-access-8nxw5\") pod \"postgres-7c7656cc6d-ns96t\" (UID: \"16adc9ab-1dfd-47ae-9617-80fe07c16107\") " pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:47:27.772006 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.771975    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-854544c579-fjhbn"]
May 25 23:47:27.775111 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.775096    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-fjhbn"
May 25 23:47:27.784967 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.784942    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-fjhbn"]
May 25 23:47:27.854485 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.854447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhr7\" (UniqueName: \"kubernetes.io/projected/739ee286-f93f-49ce-817d-4a1981d0d176-kube-api-access-qfhr7\") pod \"redis-854544c579-fjhbn\" (UID: \"739ee286-f93f-49ce-817d-4a1981d0d176\") " pod="test-ns-feast/redis-854544c579-fjhbn"
May 25 23:47:27.854670 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.854504    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/16adc9ab-1dfd-47ae-9617-80fe07c16107-postgresdata\") pod \"postgres-7c7656cc6d-ns96t\" (UID: \"16adc9ab-1dfd-47ae-9617-80fe07c16107\") " pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:47:27.854670 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.854531    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8nxw5\" (UniqueName: \"kubernetes.io/projected/16adc9ab-1dfd-47ae-9617-80fe07c16107-kube-api-access-8nxw5\") pod \"postgres-7c7656cc6d-ns96t\" (UID: \"16adc9ab-1dfd-47ae-9617-80fe07c16107\") " pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:47:27.854958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.854938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/16adc9ab-1dfd-47ae-9617-80fe07c16107-postgresdata\") pod \"postgres-7c7656cc6d-ns96t\" (UID: \"16adc9ab-1dfd-47ae-9617-80fe07c16107\") " pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:47:27.862987 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.862966    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nxw5\" (UniqueName: \"kubernetes.io/projected/16adc9ab-1dfd-47ae-9617-80fe07c16107-kube-api-access-8nxw5\") pod \"postgres-7c7656cc6d-ns96t\" (UID: \"16adc9ab-1dfd-47ae-9617-80fe07c16107\") " pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:47:27.955535 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.955492    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhr7\" (UniqueName: \"kubernetes.io/projected/739ee286-f93f-49ce-817d-4a1981d0d176-kube-api-access-qfhr7\") pod \"redis-854544c579-fjhbn\" (UID: \"739ee286-f93f-49ce-817d-4a1981d0d176\") " pod="test-ns-feast/redis-854544c579-fjhbn"
May 25 23:47:27.963797 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:27.963768    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhr7\" (UniqueName: \"kubernetes.io/projected/739ee286-f93f-49ce-817d-4a1981d0d176-kube-api-access-qfhr7\") pod \"redis-854544c579-fjhbn\" (UID: \"739ee286-f93f-49ce-817d-4a1981d0d176\") " pod="test-ns-feast/redis-854544c579-fjhbn"
May 25 23:47:28.056688 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:28.056655    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:47:28.085527 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:28.085494    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-fjhbn"
May 25 23:47:28.214779 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:28.214712    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-ns96t"]
May 25 23:47:28.218993 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:47:28.218964    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16adc9ab_1dfd_47ae_9617_80fe07c16107.slice/crio-affa029b1b6a8b54030acd4d02750dfef1a1e97c546a86bfabc96d5e3b5251c7 WatchSource:0}: Error finding container affa029b1b6a8b54030acd4d02750dfef1a1e97c546a86bfabc96d5e3b5251c7: Status 404 returned error can't find the container with id affa029b1b6a8b54030acd4d02750dfef1a1e97c546a86bfabc96d5e3b5251c7
May 25 23:47:28.220618 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:28.220601    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 23:47:28.232587 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:28.232486    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-fjhbn"]
May 25 23:47:28.235153 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:47:28.235129    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod739ee286_f93f_49ce_817d_4a1981d0d176.slice/crio-e5c0110fde7bf9e32714f1c86a90f7811b5bc90bebe1475c9fc7b9c6379e91ec WatchSource:0}: Error finding container e5c0110fde7bf9e32714f1c86a90f7811b5bc90bebe1475c9fc7b9c6379e91ec: Status 404 returned error can't find the container with id e5c0110fde7bf9e32714f1c86a90f7811b5bc90bebe1475c9fc7b9c6379e91ec
May 25 23:47:28.383062 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:28.382974    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-fjhbn" event={"ID":"739ee286-f93f-49ce-817d-4a1981d0d176","Type":"ContainerStarted","Data":"e5c0110fde7bf9e32714f1c86a90f7811b5bc90bebe1475c9fc7b9c6379e91ec"}
May 25 23:47:28.383951 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:28.383927    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-ns96t" event={"ID":"16adc9ab-1dfd-47ae-9617-80fe07c16107","Type":"ContainerStarted","Data":"affa029b1b6a8b54030acd4d02750dfef1a1e97c546a86bfabc96d5e3b5251c7"}
May 25 23:47:30.392923 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:30.392879    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-fjhbn" event={"ID":"739ee286-f93f-49ce-817d-4a1981d0d176","Type":"ContainerStarted","Data":"7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d"}
May 25 23:47:30.408276 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:30.408213    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/redis-854544c579-fjhbn" podStartSLOduration=1.699534858 podStartE2EDuration="3.408193575s" podCreationTimestamp="2026-05-25 23:47:27 +0000 UTC" firstStartedPulling="2026-05-25 23:47:28.236855442 +0000 UTC m=+747.791799682" lastFinishedPulling="2026-05-25 23:47:29.945514157 +0000 UTC m=+749.500458399" observedRunningTime="2026-05-25 23:47:30.407929111 +0000 UTC m=+749.962873373" watchObservedRunningTime="2026-05-25 23:47:30.408193575 +0000 UTC m=+749.963137839"
May 25 23:47:34.406339 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:34.406299    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-ns96t" event={"ID":"16adc9ab-1dfd-47ae-9617-80fe07c16107","Type":"ContainerStarted","Data":"46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252"}
May 25 23:47:34.421881 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:34.421824    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-ns96t" podStartSLOduration=1.893611794 podStartE2EDuration="7.421806379s" podCreationTimestamp="2026-05-25 23:47:27 +0000 UTC" firstStartedPulling="2026-05-25 23:47:28.220750286 +0000 UTC m=+747.775694525" lastFinishedPulling="2026-05-25 23:47:33.748944869 +0000 UTC m=+753.303889110" observedRunningTime="2026-05-25 23:47:34.42113849 +0000 UTC m=+753.976082753" watchObservedRunningTime="2026-05-25 23:47:34.421806379 +0000 UTC m=+753.976750641"
May 25 23:47:38.454594 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.454559    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"]
May 25 23:47:38.462648 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.462621    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.465637 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.465609    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-qlnf6\""
May 25 23:47:38.465637 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.465635    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 25 23:47:38.465839 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.465615    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 25 23:47:38.465839 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.465621    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 25 23:47:38.465839 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.465615    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 25 23:47:38.472863 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.472836    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"]
May 25 23:47:38.547608 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.547571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6696b20-e821-4f57-b202-65d2825681e6-feast-data\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.547831 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.547692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-offline-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.547831 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.547744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-online-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.547831 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.547798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbd9\" (UniqueName: \"kubernetes.io/projected/a6696b20-e821-4f57-b202-65d2825681e6-kube-api-access-5vbd9\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.547958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.547873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-registry-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.649056 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.649025    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-offline-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.649056 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.649062    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-online-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.649310 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.649082    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbd9\" (UniqueName: \"kubernetes.io/projected/a6696b20-e821-4f57-b202-65d2825681e6-kube-api-access-5vbd9\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.649310 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:38.649178    2578 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found
May 25 23:47:38.649310 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:47:38.649227    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-online-tls podName:a6696b20-e821-4f57-b202-65d2825681e6 nodeName:}" failed. No retries permitted until 2026-05-25 23:47:39.149211385 +0000 UTC m=+758.704155628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-online-tls") pod "feast-credit-scoring-744bbd9b69-5p9tk" (UID: "a6696b20-e821-4f57-b202-65d2825681e6") : secret "feast-credit-scoring-online-tls" not found
May 25 23:47:38.649310 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.649219    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-registry-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.649310 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.649257    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6696b20-e821-4f57-b202-65d2825681e6-feast-data\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.649604 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.649587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6696b20-e821-4f57-b202-65d2825681e6-feast-data\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.651599 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.651579    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-registry-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.651696 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.651598    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-offline-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:38.658632 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:38.658608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbd9\" (UniqueName: \"kubernetes.io/projected/a6696b20-e821-4f57-b202-65d2825681e6-kube-api-access-5vbd9\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:39.152851 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:39.152815    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-online-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:39.155307 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:39.155281    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-online-tls\") pod \"feast-credit-scoring-744bbd9b69-5p9tk\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") " pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:39.372192 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:39.372141    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:39.507609 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:39.507587    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"]
May 25 23:47:39.509634 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:47:39.509601    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6696b20_e821_4f57_b202_65d2825681e6.slice/crio-629961c607dd23c8904d7c2d838792bcfc3b5c514276903901469284867f37eb WatchSource:0}: Error finding container 629961c607dd23c8904d7c2d838792bcfc3b5c514276903901469284867f37eb: Status 404 returned error can't find the container with id 629961c607dd23c8904d7c2d838792bcfc3b5c514276903901469284867f37eb
May 25 23:47:40.426861 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:40.426821    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerStarted","Data":"1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e"}
May 25 23:47:40.426861 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:40.426863    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerStarted","Data":"629961c607dd23c8904d7c2d838792bcfc3b5c514276903901469284867f37eb"}
May 25 23:47:41.431258 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:41.431220    2578 generic.go:358] "Generic (PLEG): container finished" podID="a6696b20-e821-4f57-b202-65d2825681e6" containerID="1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e" exitCode=0
May 25 23:47:41.431675 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:41.431285    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerDied","Data":"1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e"}
May 25 23:47:42.436512 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:42.436476    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerStarted","Data":"40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4"}
May 25 23:47:48.454123 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:48.454088    2578 generic.go:358] "Generic (PLEG): container finished" podID="a6696b20-e821-4f57-b202-65d2825681e6" containerID="40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4" exitCode=0
May 25 23:47:48.454497 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:48.454164    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerDied","Data":"40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4"}
May 25 23:47:49.461983 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:49.461949    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerStarted","Data":"6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c"}
May 25 23:47:49.462460 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:49.461996    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerStarted","Data":"96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74"}
May 25 23:47:49.462460 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:49.462011    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerStarted","Data":"506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5"}
May 25 23:47:49.490175 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:49.490117    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podStartSLOduration=11.490093698 podStartE2EDuration="11.490093698s" podCreationTimestamp="2026-05-25 23:47:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:47:49.488147296 +0000 UTC m=+769.043091567" watchObservedRunningTime="2026-05-25 23:47:49.490093698 +0000 UTC m=+769.045037962"
May 25 23:47:51.373005 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:51.372942    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:51.373005 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:51.372995    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:51.373005 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:51.373009    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:51.374748 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:51.374684    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.22:6571: connect: connection refused"
May 25 23:47:51.374748 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:51.374702    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.22:8816: connect: connection refused"
May 25 23:47:51.374748 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:51.374716    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused"
May 25 23:47:54.374263 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:54.374164    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.22:6571: connect: connection refused"
May 25 23:47:54.375940 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:54.375917    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:54.377319 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:54.377301    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:54.378886 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:54.378870    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:54.717472 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:54.717372    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:55.482515 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:55.482482    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:55.486616 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:55.486593    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:47:57.373587 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:57.373540    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.22:6571: connect: connection refused"
May 25 23:47:59.372564 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:47:59.372517    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:48:00.373004 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:48:00.372970    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:48:00.373746 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:48:00.373706    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:49:04.802207 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:04.802172    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"]
May 25 23:49:04.802698 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:04.802523    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="registry" containerID="cri-o://506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5" gracePeriod=30
May 25 23:49:04.802698 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:04.802561    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="offline" containerID="cri-o://6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c" gracePeriod=30
May 25 23:49:04.802698 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:04.802565    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="online" containerID="cri-o://96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74" gracePeriod=30
May 25 23:49:04.818886 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:04.818844    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-ns96t"]
May 25 23:49:04.819069 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:04.819049    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-ns96t" podUID="16adc9ab-1dfd-47ae-9617-80fe07c16107" containerName="postgres" containerID="cri-o://46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252" gracePeriod=30
May 25 23:49:04.823377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:04.823348    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-fjhbn"]
May 25 23:49:04.823573 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:04.823552    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/redis-854544c579-fjhbn" podUID="739ee286-f93f-49ce-817d-4a1981d0d176" containerName="redis" containerID="cri-o://7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d" gracePeriod=30
May 25 23:49:05.088777 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.088753    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:49:05.092148 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.092125    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-fjhbn"
May 25 23:49:05.176154 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.176118    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nxw5\" (UniqueName: \"kubernetes.io/projected/16adc9ab-1dfd-47ae-9617-80fe07c16107-kube-api-access-8nxw5\") pod \"16adc9ab-1dfd-47ae-9617-80fe07c16107\" (UID: \"16adc9ab-1dfd-47ae-9617-80fe07c16107\") "
May 25 23:49:05.176308 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.176196    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfhr7\" (UniqueName: \"kubernetes.io/projected/739ee286-f93f-49ce-817d-4a1981d0d176-kube-api-access-qfhr7\") pod \"739ee286-f93f-49ce-817d-4a1981d0d176\" (UID: \"739ee286-f93f-49ce-817d-4a1981d0d176\") "
May 25 23:49:05.176308 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.176225    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/16adc9ab-1dfd-47ae-9617-80fe07c16107-postgresdata\") pod \"16adc9ab-1dfd-47ae-9617-80fe07c16107\" (UID: \"16adc9ab-1dfd-47ae-9617-80fe07c16107\") "
May 25 23:49:05.183768 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.183695    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739ee286-f93f-49ce-817d-4a1981d0d176-kube-api-access-qfhr7" (OuterVolumeSpecName: "kube-api-access-qfhr7") pod "739ee286-f93f-49ce-817d-4a1981d0d176" (UID: "739ee286-f93f-49ce-817d-4a1981d0d176"). InnerVolumeSpecName "kube-api-access-qfhr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:49:05.184916 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.184884    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16adc9ab-1dfd-47ae-9617-80fe07c16107-kube-api-access-8nxw5" (OuterVolumeSpecName: "kube-api-access-8nxw5") pod "16adc9ab-1dfd-47ae-9617-80fe07c16107" (UID: "16adc9ab-1dfd-47ae-9617-80fe07c16107"). InnerVolumeSpecName "kube-api-access-8nxw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:49:05.208971 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.208932    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16adc9ab-1dfd-47ae-9617-80fe07c16107-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "16adc9ab-1dfd-47ae-9617-80fe07c16107" (UID: "16adc9ab-1dfd-47ae-9617-80fe07c16107"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:49:05.278463 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.278424    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nxw5\" (UniqueName: \"kubernetes.io/projected/16adc9ab-1dfd-47ae-9617-80fe07c16107-kube-api-access-8nxw5\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:49:05.278463 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.278459    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfhr7\" (UniqueName: \"kubernetes.io/projected/739ee286-f93f-49ce-817d-4a1981d0d176-kube-api-access-qfhr7\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:49:05.278463 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.278473    2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/16adc9ab-1dfd-47ae-9617-80fe07c16107-postgresdata\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:49:05.483219 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.483118    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused"
May 25 23:49:05.700910 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.700876    2578 generic.go:358] "Generic (PLEG): container finished" podID="739ee286-f93f-49ce-817d-4a1981d0d176" containerID="7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d" exitCode=0
May 25 23:49:05.701118 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.700936    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-fjhbn"
May 25 23:49:05.701118 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.700975    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-fjhbn" event={"ID":"739ee286-f93f-49ce-817d-4a1981d0d176","Type":"ContainerDied","Data":"7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d"}
May 25 23:49:05.701118 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.701010    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-fjhbn" event={"ID":"739ee286-f93f-49ce-817d-4a1981d0d176","Type":"ContainerDied","Data":"e5c0110fde7bf9e32714f1c86a90f7811b5bc90bebe1475c9fc7b9c6379e91ec"}
May 25 23:49:05.701118 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.701028    2578 scope.go:117] "RemoveContainer" containerID="7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d"
May 25 23:49:05.703615 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.703592    2578 generic.go:358] "Generic (PLEG): container finished" podID="a6696b20-e821-4f57-b202-65d2825681e6" containerID="96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74" exitCode=0
May 25 23:49:05.703767 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.703673    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerDied","Data":"96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74"}
May 25 23:49:05.704948 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.704918    2578 generic.go:358] "Generic (PLEG): container finished" podID="16adc9ab-1dfd-47ae-9617-80fe07c16107" containerID="46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252" exitCode=0
May 25 23:49:05.705047 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.704990    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-ns96t" event={"ID":"16adc9ab-1dfd-47ae-9617-80fe07c16107","Type":"ContainerDied","Data":"46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252"}
May 25 23:49:05.705047 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.704998    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-ns96t"
May 25 23:49:05.705047 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.705014    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-ns96t" event={"ID":"16adc9ab-1dfd-47ae-9617-80fe07c16107","Type":"ContainerDied","Data":"affa029b1b6a8b54030acd4d02750dfef1a1e97c546a86bfabc96d5e3b5251c7"}
May 25 23:49:05.711028 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.711005    2578 scope.go:117] "RemoveContainer" containerID="7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d"
May 25 23:49:05.711316 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:49:05.711296    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d\": container with ID starting with 7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d not found: ID does not exist" containerID="7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d"
May 25 23:49:05.711364 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.711330    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d"} err="failed to get container status \"7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d\": rpc error: code = NotFound desc = could not find container \"7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d\": container with ID starting with 7a57748156f73f124f98f90fc82a31488f9c8ccc9fcdd68a128e5e95fdb35e5d not found: ID does not exist"
May 25 23:49:05.711364 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.711351    2578 scope.go:117] "RemoveContainer" containerID="46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252"
May 25 23:49:05.718699 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.718679    2578 scope.go:117] "RemoveContainer" containerID="46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252"
May 25 23:49:05.719059 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:49:05.719037    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252\": container with ID starting with 46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252 not found: ID does not exist" containerID="46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252"
May 25 23:49:05.719109 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.719071    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252"} err="failed to get container status \"46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252\": rpc error: code = NotFound desc = could not find container \"46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252\": container with ID starting with 46f7584c4dd4c80008f5f5257ae78e5e087c30a5e9a0527d779273a7ee9e9252 not found: ID does not exist"
May 25 23:49:05.721970 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.721947    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-fjhbn"]
May 25 23:49:05.725003 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.724983    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-854544c579-fjhbn"]
May 25 23:49:05.735282 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.735214    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-ns96t"]
May 25 23:49:05.740916 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:05.740889    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-ns96t"]
May 25 23:49:07.008468 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:07.008433    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16adc9ab-1dfd-47ae-9617-80fe07c16107" path="/var/lib/kubelet/pods/16adc9ab-1dfd-47ae-9617-80fe07c16107/volumes"
May 25 23:49:07.008906 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:07.008774    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="739ee286-f93f-49ce-817d-4a1981d0d176" path="/var/lib/kubelet/pods/739ee286-f93f-49ce-817d-4a1981d0d176/volumes"
May 25 23:49:15.483391 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:15.483341    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused"
May 25 23:49:25.482789 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:25.482717    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused"
May 25 23:49:25.483203 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:25.482869    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:49:34.984285 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:34.984260    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:49:35.032112 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.032078    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6696b20-e821-4f57-b202-65d2825681e6-feast-data\") pod \"a6696b20-e821-4f57-b202-65d2825681e6\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") "
May 25 23:49:35.032112 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.032118    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-offline-tls\") pod \"a6696b20-e821-4f57-b202-65d2825681e6\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") "
May 25 23:49:35.032374 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.032142    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-registry-tls\") pod \"a6696b20-e821-4f57-b202-65d2825681e6\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") "
May 25 23:49:35.032374 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.032186    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vbd9\" (UniqueName: \"kubernetes.io/projected/a6696b20-e821-4f57-b202-65d2825681e6-kube-api-access-5vbd9\") pod \"a6696b20-e821-4f57-b202-65d2825681e6\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") "
May 25 23:49:35.032374 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.032259    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-online-tls\") pod \"a6696b20-e821-4f57-b202-65d2825681e6\" (UID: \"a6696b20-e821-4f57-b202-65d2825681e6\") "
May 25 23:49:35.034373 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.034332    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a6696b20-e821-4f57-b202-65d2825681e6" (UID: "a6696b20-e821-4f57-b202-65d2825681e6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:49:35.034503 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.034388    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a6696b20-e821-4f57-b202-65d2825681e6" (UID: "a6696b20-e821-4f57-b202-65d2825681e6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:49:35.034645 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.034628    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-online-tls" (OuterVolumeSpecName: "online-tls") pod "a6696b20-e821-4f57-b202-65d2825681e6" (UID: "a6696b20-e821-4f57-b202-65d2825681e6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:49:35.034702 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.034670    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6696b20-e821-4f57-b202-65d2825681e6-kube-api-access-5vbd9" (OuterVolumeSpecName: "kube-api-access-5vbd9") pod "a6696b20-e821-4f57-b202-65d2825681e6" (UID: "a6696b20-e821-4f57-b202-65d2825681e6"). InnerVolumeSpecName "kube-api-access-5vbd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:49:35.040461 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.040432    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6696b20-e821-4f57-b202-65d2825681e6-feast-data" (OuterVolumeSpecName: "feast-data") pod "a6696b20-e821-4f57-b202-65d2825681e6" (UID: "a6696b20-e821-4f57-b202-65d2825681e6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:49:35.133066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.132963    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vbd9\" (UniqueName: \"kubernetes.io/projected/a6696b20-e821-4f57-b202-65d2825681e6-kube-api-access-5vbd9\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:49:35.133066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.133008    2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-online-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:49:35.133066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.133019    2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6696b20-e821-4f57-b202-65d2825681e6-feast-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:49:35.133066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.133027    2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-offline-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:49:35.133066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.133036    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6696b20-e821-4f57-b202-65d2825681e6-registry-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:49:35.799507 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.799466    2578 generic.go:358] "Generic (PLEG): container finished" podID="a6696b20-e821-4f57-b202-65d2825681e6" containerID="6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c" exitCode=137
May 25 23:49:35.799507 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.799495    2578 generic.go:358] "Generic (PLEG): container finished" podID="a6696b20-e821-4f57-b202-65d2825681e6" containerID="506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5" exitCode=137
May 25 23:49:35.799799 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.799536    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerDied","Data":"6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c"}
May 25 23:49:35.799799 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.799570    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerDied","Data":"506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5"}
May 25 23:49:35.799799 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.799584    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk" event={"ID":"a6696b20-e821-4f57-b202-65d2825681e6","Type":"ContainerDied","Data":"629961c607dd23c8904d7c2d838792bcfc3b5c514276903901469284867f37eb"}
May 25 23:49:35.799799 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.799589    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"
May 25 23:49:35.799799 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.799602    2578 scope.go:117] "RemoveContainer" containerID="6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c"
May 25 23:49:35.808967 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.808801    2578 scope.go:117] "RemoveContainer" containerID="96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74"
May 25 23:49:35.818037 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.818016    2578 scope.go:117] "RemoveContainer" containerID="506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5"
May 25 23:49:35.825007 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.824987    2578 scope.go:117] "RemoveContainer" containerID="40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4"
May 25 23:49:35.831651 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.831631    2578 scope.go:117] "RemoveContainer" containerID="1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e"
May 25 23:49:35.838328 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.838309    2578 scope.go:117] "RemoveContainer" containerID="6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c"
May 25 23:49:35.838601 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:49:35.838573    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c\": container with ID starting with 6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c not found: ID does not exist" containerID="6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c"
May 25 23:49:35.838682 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.838612    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c"} err="failed to get container status \"6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c\": rpc error: code = NotFound desc = could not find container \"6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c\": container with ID starting with 6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c not found: ID does not exist"
May 25 23:49:35.838682 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.838633    2578 scope.go:117] "RemoveContainer" containerID="96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74"
May 25 23:49:35.838910 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:49:35.838895    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74\": container with ID starting with 96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74 not found: ID does not exist" containerID="96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74"
May 25 23:49:35.838961 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.838923    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74"} err="failed to get container status \"96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74\": rpc error: code = NotFound desc = could not find container \"96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74\": container with ID starting with 96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74 not found: ID does not exist"
May 25 23:49:35.838961 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.838938    2578 scope.go:117] "RemoveContainer" containerID="506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5"
May 25 23:49:35.839181 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:49:35.839165    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5\": container with ID starting with 506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5 not found: ID does not exist" containerID="506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5"
May 25 23:49:35.839226 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.839184    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5"} err="failed to get container status \"506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5\": rpc error: code = NotFound desc = could not find container \"506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5\": container with ID starting with 506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5 not found: ID does not exist"
May 25 23:49:35.839226 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.839203    2578 scope.go:117] "RemoveContainer" containerID="40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4"
May 25 23:49:35.839443 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:49:35.839426    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4\": container with ID starting with 40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4 not found: ID does not exist" containerID="40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4"
May 25 23:49:35.839482 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.839449    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4"} err="failed to get container status \"40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4\": rpc error: code = NotFound desc = could not find container \"40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4\": container with ID starting with 40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4 not found: ID does not exist"
May 25 23:49:35.839482 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.839464    2578 scope.go:117] "RemoveContainer" containerID="1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e"
May 25 23:49:35.839684 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:49:35.839665    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e\": container with ID starting with 1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e not found: ID does not exist" containerID="1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e"
May 25 23:49:35.839776 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.839687    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e"} err="failed to get container status \"1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e\": rpc error: code = NotFound desc = could not find container \"1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e\": container with ID starting with 1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e not found: ID does not exist"
May 25 23:49:35.839776 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.839701    2578 scope.go:117] "RemoveContainer" containerID="6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c"
May 25 23:49:35.840057 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.840028    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c"} err="failed to get container status \"6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c\": rpc error: code = NotFound desc = could not find container \"6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c\": container with ID starting with 6fa06da0f0b451dd73813e28f1f166d69231b9ba85fea0903c32998d79dfea6c not found: ID does not exist"
May 25 23:49:35.840057 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.840053    2578 scope.go:117] "RemoveContainer" containerID="96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74"
May 25 23:49:35.840297 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.840278    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74"} err="failed to get container status \"96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74\": rpc error: code = NotFound desc = could not find container \"96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74\": container with ID starting with 96e77d92ee85aef0d853fea7f82a4396f36187be01317c9eba1b6a8bf066fc74 not found: ID does not exist"
May 25 23:49:35.840297 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.840297    2578 scope.go:117] "RemoveContainer" containerID="506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5"
May 25 23:49:35.840569 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.840541    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5"} err="failed to get container status \"506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5\": rpc error: code = NotFound desc = could not find container \"506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5\": container with ID starting with 506b541c461946e7289c40be137ef7aaf7b0226039fff22518fead565c1436c5 not found: ID does not exist"
May 25 23:49:35.840656 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.840569    2578 scope.go:117] "RemoveContainer" containerID="40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4"
May 25 23:49:35.840868 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.840847    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4"} err="failed to get container status \"40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4\": rpc error: code = NotFound desc = could not find container \"40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4\": container with ID starting with 40c76036a711ae4a6a25c6ecca8c933173255c30e1ab05769e919ee56074f4c4 not found: ID does not exist"
May 25 23:49:35.840868 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.840865    2578 scope.go:117] "RemoveContainer" containerID="1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e"
May 25 23:49:35.841243 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.841217    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e"} err="failed to get container status \"1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e\": rpc error: code = NotFound desc = could not find container \"1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e\": container with ID starting with 1f16f84a0ea93290b1c8f19b0ceab6e5f20700f000b48428c8f60b498c51ce4e not found: ID does not exist"
May 25 23:49:35.844076 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.844054    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"]
May 25 23:49:35.849608 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:35.849585    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-744bbd9b69-5p9tk"]
May 25 23:49:37.007315 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:49:37.007278    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6696b20-e821-4f57-b202-65d2825681e6" path="/var/lib/kubelet/pods/a6696b20-e821-4f57-b202-65d2825681e6/volumes"
May 25 23:50:00.970162 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:50:00.970066    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:50:00.971822 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:50:00.971800    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:51:32.216133 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216044    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-plkg2"]
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216499    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="feast-apply"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216520    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="feast-apply"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216532    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="registry"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216540    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="registry"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216551    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="offline"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216558    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="offline"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216574    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="online"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216582    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="online"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216596    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16adc9ab-1dfd-47ae-9617-80fe07c16107" containerName="postgres"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216605    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="16adc9ab-1dfd-47ae-9617-80fe07c16107" containerName="postgres"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216632    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="739ee286-f93f-49ce-817d-4a1981d0d176" containerName="redis"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216640    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="739ee286-f93f-49ce-817d-4a1981d0d176" containerName="redis"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216660    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="feast-init"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216668    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="feast-init"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216761    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="739ee286-f93f-49ce-817d-4a1981d0d176" containerName="redis"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216773    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="16adc9ab-1dfd-47ae-9617-80fe07c16107" containerName="postgres"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216784    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="offline"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216799    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="registry"
May 25 23:51:32.218552 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.216809    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a6696b20-e821-4f57-b202-65d2825681e6" containerName="online"
May 25 23:51:32.219944 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.219923    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-plkg2"
May 25 23:51:32.222570 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.222544    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 25 23:51:32.222815 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.222597    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 25 23:51:32.223500 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.223480    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-d5qdn\""
May 25 23:51:32.235941 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.235907    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-plkg2"]
May 25 23:51:32.275414 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.275383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8tpl\" (UniqueName: \"kubernetes.io/projected/8ea94386-527a-4d99-acfa-18dd22657443-kube-api-access-k8tpl\") pod \"redis-854544c579-plkg2\" (UID: \"8ea94386-527a-4d99-acfa-18dd22657443\") " pod="test-ns-feast-rest/redis-854544c579-plkg2"
May 25 23:51:32.376576 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.376538    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k8tpl\" (UniqueName: \"kubernetes.io/projected/8ea94386-527a-4d99-acfa-18dd22657443-kube-api-access-k8tpl\") pod \"redis-854544c579-plkg2\" (UID: \"8ea94386-527a-4d99-acfa-18dd22657443\") " pod="test-ns-feast-rest/redis-854544c579-plkg2"
May 25 23:51:32.385196 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.385168    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8tpl\" (UniqueName: \"kubernetes.io/projected/8ea94386-527a-4d99-acfa-18dd22657443-kube-api-access-k8tpl\") pod \"redis-854544c579-plkg2\" (UID: \"8ea94386-527a-4d99-acfa-18dd22657443\") " pod="test-ns-feast-rest/redis-854544c579-plkg2"
May 25 23:51:32.542222 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.542122    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-plkg2"
May 25 23:51:32.670927 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:32.670777    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-plkg2"]
May 25 23:51:32.673468 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:51:32.673438    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ea94386_527a_4d99_acfa_18dd22657443.slice/crio-d1c0c1cdf8ab438493999c28550385eb4c11e610b0f367855d78935ab8cf65d2 WatchSource:0}: Error finding container d1c0c1cdf8ab438493999c28550385eb4c11e610b0f367855d78935ab8cf65d2: Status 404 returned error can't find the container with id d1c0c1cdf8ab438493999c28550385eb4c11e610b0f367855d78935ab8cf65d2
May 25 23:51:33.128863 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:33.128827    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-plkg2" event={"ID":"8ea94386-527a-4d99-acfa-18dd22657443","Type":"ContainerStarted","Data":"836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425"}
May 25 23:51:33.128863 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:33.128864    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-plkg2" event={"ID":"8ea94386-527a-4d99-acfa-18dd22657443","Type":"ContainerStarted","Data":"d1c0c1cdf8ab438493999c28550385eb4c11e610b0f367855d78935ab8cf65d2"}
May 25 23:51:33.145987 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:33.145934    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-plkg2" podStartSLOduration=1.145914603 podStartE2EDuration="1.145914603s" podCreationTimestamp="2026-05-25 23:51:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:51:33.144482069 +0000 UTC m=+992.699426333" watchObservedRunningTime="2026-05-25 23:51:33.145914603 +0000 UTC m=+992.700858861"
May 25 23:51:37.635605 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.635569    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"]
May 25 23:51:37.638877 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.638854    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:51:37.641337 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.641305    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 25 23:51:37.648448 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.648423    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"]
May 25 23:51:37.719367 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.719315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5d629baa-d80b-42ad-83ca-f36d2f444297-postgresdata\") pod \"postgres-7c7656cc6d-zmzq7\" (UID: \"5d629baa-d80b-42ad-83ca-f36d2f444297\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:51:37.719367 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.719374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6cnm\" (UniqueName: \"kubernetes.io/projected/5d629baa-d80b-42ad-83ca-f36d2f444297-kube-api-access-s6cnm\") pod \"postgres-7c7656cc6d-zmzq7\" (UID: \"5d629baa-d80b-42ad-83ca-f36d2f444297\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:51:37.820689 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.820645    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5d629baa-d80b-42ad-83ca-f36d2f444297-postgresdata\") pod \"postgres-7c7656cc6d-zmzq7\" (UID: \"5d629baa-d80b-42ad-83ca-f36d2f444297\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:51:37.820867 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.820704    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s6cnm\" (UniqueName: \"kubernetes.io/projected/5d629baa-d80b-42ad-83ca-f36d2f444297-kube-api-access-s6cnm\") pod \"postgres-7c7656cc6d-zmzq7\" (UID: \"5d629baa-d80b-42ad-83ca-f36d2f444297\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:51:37.821070 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.821050    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5d629baa-d80b-42ad-83ca-f36d2f444297-postgresdata\") pod \"postgres-7c7656cc6d-zmzq7\" (UID: \"5d629baa-d80b-42ad-83ca-f36d2f444297\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:51:37.829392 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.829359    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6cnm\" (UniqueName: \"kubernetes.io/projected/5d629baa-d80b-42ad-83ca-f36d2f444297-kube-api-access-s6cnm\") pod \"postgres-7c7656cc6d-zmzq7\" (UID: \"5d629baa-d80b-42ad-83ca-f36d2f444297\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:51:37.949389 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:37.949303    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:51:38.071958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:38.071932    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"]
May 25 23:51:38.074511 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:51:38.074474    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d629baa_d80b_42ad_83ca_f36d2f444297.slice/crio-3b08e1278b6cd5faedda1914b24fe591d25fe392a2dc9c6997f5521e442e037a WatchSource:0}: Error finding container 3b08e1278b6cd5faedda1914b24fe591d25fe392a2dc9c6997f5521e442e037a: Status 404 returned error can't find the container with id 3b08e1278b6cd5faedda1914b24fe591d25fe392a2dc9c6997f5521e442e037a
May 25 23:51:38.144341 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:38.144303    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7" event={"ID":"5d629baa-d80b-42ad-83ca-f36d2f444297","Type":"ContainerStarted","Data":"3b08e1278b6cd5faedda1914b24fe591d25fe392a2dc9c6997f5521e442e037a"}
May 25 23:51:39.149540 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:39.149503    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7" event={"ID":"5d629baa-d80b-42ad-83ca-f36d2f444297","Type":"ContainerStarted","Data":"f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c"}
May 25 23:51:39.166421 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:39.166152    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7" podStartSLOduration=2.166134135 podStartE2EDuration="2.166134135s" podCreationTimestamp="2026-05-25 23:51:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:51:39.165445081 +0000 UTC m=+998.720389342" watchObservedRunningTime="2026-05-25 23:51:39.166134135 +0000 UTC m=+998.721078397"
May 25 23:51:43.363469 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.363420    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"]
May 25 23:51:43.366924 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.366899    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.369775 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.369751    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 25 23:51:43.369908 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.369801    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 25 23:51:43.370036 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.370021    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 25 23:51:43.370082 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.370038    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-5jfsg\""
May 25 23:51:43.382774 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.382710    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"]
May 25 23:51:43.465701 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.465669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg9zk\" (UniqueName: \"kubernetes.io/projected/522a6a19-f6de-4468-8c15-69881db6961f-kube-api-access-cg9zk\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.465901 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.465745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/522a6a19-f6de-4468-8c15-69881db6961f-feast-data\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.465901 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.465820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-online-tls\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.465901 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.465854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-registry-tls\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.566907 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.566865    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/522a6a19-f6de-4468-8c15-69881db6961f-feast-data\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.567071 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.566938    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-online-tls\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.567071 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.566966    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-registry-tls\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.567071 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.567012    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cg9zk\" (UniqueName: \"kubernetes.io/projected/522a6a19-f6de-4468-8c15-69881db6961f-kube-api-access-cg9zk\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.567175 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:51:43.567094    2578 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found
May 25 23:51:43.567175 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:51:43.567171    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-online-tls podName:522a6a19-f6de-4468-8c15-69881db6961f nodeName:}" failed. No retries permitted until 2026-05-25 23:51:44.06715363 +0000 UTC m=+1003.622097874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-online-tls") pod "feast-credit-scoring-984d96475-2t4gz" (UID: "522a6a19-f6de-4468-8c15-69881db6961f") : secret "feast-credit-scoring-online-tls" not found
May 25 23:51:43.567242 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:51:43.567096    2578 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found
May 25 23:51:43.567274 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:51:43.567245    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-registry-tls podName:522a6a19-f6de-4468-8c15-69881db6961f nodeName:}" failed. No retries permitted until 2026-05-25 23:51:44.067227767 +0000 UTC m=+1003.622172023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-registry-tls") pod "feast-credit-scoring-984d96475-2t4gz" (UID: "522a6a19-f6de-4468-8c15-69881db6961f") : secret "feast-credit-scoring-registry-tls" not found
May 25 23:51:43.567319 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.567306    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/522a6a19-f6de-4468-8c15-69881db6961f-feast-data\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:43.578406 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:43.578373    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg9zk\" (UniqueName: \"kubernetes.io/projected/522a6a19-f6de-4468-8c15-69881db6961f-kube-api-access-cg9zk\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:44.070833 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:44.070772    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-online-tls\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:44.070833 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:44.070839    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-registry-tls\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:44.073403 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:44.073377    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-registry-tls\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:44.073403 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:44.073389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-online-tls\") pod \"feast-credit-scoring-984d96475-2t4gz\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") " pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:44.276589 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:44.276535    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:44.414437 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:44.414396    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"]
May 25 23:51:44.419470 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:51:44.419439    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522a6a19_f6de_4468_8c15_69881db6961f.slice/crio-00f0ca9ebabbd3ec4d85f1a3c0ee24c7f203e5aa28324ab1d0866a529710146e WatchSource:0}: Error finding container 00f0ca9ebabbd3ec4d85f1a3c0ee24c7f203e5aa28324ab1d0866a529710146e: Status 404 returned error can't find the container with id 00f0ca9ebabbd3ec4d85f1a3c0ee24c7f203e5aa28324ab1d0866a529710146e
May 25 23:51:45.170639 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:45.170603    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerStarted","Data":"bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3"}
May 25 23:51:45.170639 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:45.170640    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerStarted","Data":"00f0ca9ebabbd3ec4d85f1a3c0ee24c7f203e5aa28324ab1d0866a529710146e"}
May 25 23:51:46.175143 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:46.175099    2578 generic.go:358] "Generic (PLEG): container finished" podID="522a6a19-f6de-4468-8c15-69881db6961f" containerID="bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3" exitCode=0
May 25 23:51:46.175538 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:46.175166    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerDied","Data":"bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3"}
May 25 23:51:47.180358 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:47.180318    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerStarted","Data":"dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4"}
May 25 23:51:54.201401 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:54.201366    2578 generic.go:358] "Generic (PLEG): container finished" podID="522a6a19-f6de-4468-8c15-69881db6961f" containerID="dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4" exitCode=0
May 25 23:51:54.201848 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:54.201441    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerDied","Data":"dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4"}
May 25 23:51:55.206924 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:55.206882    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerStarted","Data":"c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e"}
May 25 23:51:55.206924 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:55.206932    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerStarted","Data":"d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624"}
May 25 23:51:55.238109 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:55.238060    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podStartSLOduration=12.238045028 podStartE2EDuration="12.238045028s" podCreationTimestamp="2026-05-25 23:51:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:51:55.236286191 +0000 UTC m=+1014.791230453" watchObservedRunningTime="2026-05-25 23:51:55.238045028 +0000 UTC m=+1014.792989288"
May 25 23:51:56.277279 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:56.277236    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:56.277779 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:56.277293    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:51:56.278677 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:56.278647    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused"
May 25 23:51:56.278832 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:56.278691    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 25 23:51:59.277021 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:59.276966    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused"
May 25 23:51:59.277417 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:51:59.277037    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 25 23:52:02.276928 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:02.276879    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused"
May 25 23:52:02.282554 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:02.282519    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:52:02.283058 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:02.283033    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:52:02.286186 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:02.286156    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:52:04.277167 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:04.277135    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:52:05.277737 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:05.277668    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused"
May 25 23:52:08.277584 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:08.277552    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:52:08.278377 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:08.278357    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:52:09.404338 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.404303    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"]
May 25 23:52:09.409368 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.409347    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.412682 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.412655    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 25 23:52:09.412848 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.412830    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 25 23:52:09.412951 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.412935    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-gcmzm\""
May 25 23:52:09.413270 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.413255    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 25 23:52:09.423294 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.423271    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"]
May 25 23:52:09.518134 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.518095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-registry-tls\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.518134 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.518137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-online-tls\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.518363 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.518199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f64e3865-c764-4d7f-b577-3fa70762b31e-feast-data\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.518363 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.518252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhp2\" (UniqueName: \"kubernetes.io/projected/f64e3865-c764-4d7f-b577-3fa70762b31e-kube-api-access-5xhp2\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.619785 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.619747    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-registry-tls\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.619785 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.619795    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-online-tls\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.620052 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.619826    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f64e3865-c764-4d7f-b577-3fa70762b31e-feast-data\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.620052 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.619867    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhp2\" (UniqueName: \"kubernetes.io/projected/f64e3865-c764-4d7f-b577-3fa70762b31e-kube-api-access-5xhp2\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.620274 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.620252    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f64e3865-c764-4d7f-b577-3fa70762b31e-feast-data\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.622222 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.622201    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-online-tls\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.622324 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.622231    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-registry-tls\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.629865 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.629838    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhp2\" (UniqueName: \"kubernetes.io/projected/f64e3865-c764-4d7f-b577-3fa70762b31e-kube-api-access-5xhp2\") pod \"feast-driver-ranking-8dcdfbc44-knswv\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") " pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.718974 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.718879    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:09.862502 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:09.862473    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"]
May 25 23:52:09.864167 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:52:09.864132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64e3865_c764_4d7f_b577_3fa70762b31e.slice/crio-b3904b2423d8b4c6a8a40bb532870b3017cdcf738579c6c5e0f38205f26d2c71 WatchSource:0}: Error finding container b3904b2423d8b4c6a8a40bb532870b3017cdcf738579c6c5e0f38205f26d2c71: Status 404 returned error can't find the container with id b3904b2423d8b4c6a8a40bb532870b3017cdcf738579c6c5e0f38205f26d2c71
May 25 23:52:10.257837 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:10.257802    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerStarted","Data":"d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224"}
May 25 23:52:10.258027 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:10.257848    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerStarted","Data":"b3904b2423d8b4c6a8a40bb532870b3017cdcf738579c6c5e0f38205f26d2c71"}
May 25 23:52:15.275141 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:15.275109    2578 generic.go:358] "Generic (PLEG): container finished" podID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerID="d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224" exitCode=0
May 25 23:52:15.275555 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:15.275179    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerDied","Data":"d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224"}
May 25 23:52:16.280641 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:16.280599    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerStarted","Data":"6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6"}
May 25 23:52:23.310773 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:23.310715    2578 generic.go:358] "Generic (PLEG): container finished" podID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerID="6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6" exitCode=0
May 25 23:52:23.311155 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:23.310791    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerDied","Data":"6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6"}
May 25 23:52:24.316376 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:24.316340    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerStarted","Data":"d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce"}
May 25 23:52:24.316922 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:24.316387    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerStarted","Data":"c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c"}
May 25 23:52:24.340871 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:24.340809    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podStartSLOduration=15.34077966 podStartE2EDuration="15.34077966s" podCreationTimestamp="2026-05-25 23:52:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:52:24.33966987 +0000 UTC m=+1043.894614131" watchObservedRunningTime="2026-05-25 23:52:24.34077966 +0000 UTC m=+1043.895723922"
May 25 23:52:24.719203 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:24.719104    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:24.719203 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:24.719162    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:24.720867 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:24.720835    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 25 23:52:24.720867 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:24.720849    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.26:6571: connect: connection refused"
May 25 23:52:27.720193 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:27.720145    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.26:6571: connect: connection refused"
May 25 23:52:27.720693 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:27.720157    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 25 23:52:29.719485 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:29.719429    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:29.719882 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:29.719496    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:30.720131 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:30.720073    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.26:6571: connect: connection refused"
May 25 23:52:30.725513 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:30.725484    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:30.728833 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:30.728805    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:33.719800 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:33.719748    2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.26:6571: connect: connection refused"
May 25 23:52:36.719595 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:36.719559    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:52:36.720254 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:52:36.720235    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:53:54.361400 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.361364    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"]
May 25 23:53:54.361883 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.361586    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7" podUID="80473cc5-d469-4fdb-a1a4-8e04be8e7c8a" containerName="manager" containerID="cri-o://a982ac2d2965ec33397e45b50124e1f38618f4a9b14fc8ba277651feb0db1f07" gracePeriod=10
May 25 23:53:54.595455 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.595416    2578 generic.go:358] "Generic (PLEG): container finished" podID="80473cc5-d469-4fdb-a1a4-8e04be8e7c8a" containerID="a982ac2d2965ec33397e45b50124e1f38618f4a9b14fc8ba277651feb0db1f07" exitCode=0
May 25 23:53:54.595619 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.595492    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7" event={"ID":"80473cc5-d469-4fdb-a1a4-8e04be8e7c8a","Type":"ContainerDied","Data":"a982ac2d2965ec33397e45b50124e1f38618f4a9b14fc8ba277651feb0db1f07"}
May 25 23:53:54.624106 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.624080    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"
May 25 23:53:54.637690 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.637661    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg4l5\" (UniqueName: \"kubernetes.io/projected/80473cc5-d469-4fdb-a1a4-8e04be8e7c8a-kube-api-access-fg4l5\") pod \"80473cc5-d469-4fdb-a1a4-8e04be8e7c8a\" (UID: \"80473cc5-d469-4fdb-a1a4-8e04be8e7c8a\") "
May 25 23:53:54.640358 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.640328    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80473cc5-d469-4fdb-a1a4-8e04be8e7c8a-kube-api-access-fg4l5" (OuterVolumeSpecName: "kube-api-access-fg4l5") pod "80473cc5-d469-4fdb-a1a4-8e04be8e7c8a" (UID: "80473cc5-d469-4fdb-a1a4-8e04be8e7c8a"). InnerVolumeSpecName "kube-api-access-fg4l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:53:54.654930 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.654892    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"]
May 25 23:53:54.655369 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.655308    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="registry" containerID="cri-o://d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624" gracePeriod=30
May 25 23:53:54.655577 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.655545    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="online" containerID="cri-o://c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e" gracePeriod=30
May 25 23:53:54.663297 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.663266    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"]
May 25 23:53:54.663650 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.663618    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="registry" containerID="cri-o://c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c" gracePeriod=30
May 25 23:53:54.663876 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.663673    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="online" containerID="cri-o://d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce" gracePeriod=30
May 25 23:53:54.738712 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:54.738670    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg4l5\" (UniqueName: \"kubernetes.io/projected/80473cc5-d469-4fdb-a1a4-8e04be8e7c8a-kube-api-access-fg4l5\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:53:55.600682 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:55.600647    2578 generic.go:358] "Generic (PLEG): container finished" podID="522a6a19-f6de-4468-8c15-69881db6961f" containerID="c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e" exitCode=0
May 25 23:53:55.601182 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:55.600744    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerDied","Data":"c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e"}
May 25 23:53:55.601889 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:55.601861    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7" event={"ID":"80473cc5-d469-4fdb-a1a4-8e04be8e7c8a","Type":"ContainerDied","Data":"4c4e3dcafabe549e5cb6bd8f2b14fa77ab6d26f631fe4ffdd1702e21f232cbbf"}
May 25 23:53:55.602022 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:55.601901    2578 scope.go:117] "RemoveContainer" containerID="a982ac2d2965ec33397e45b50124e1f38618f4a9b14fc8ba277651feb0db1f07"
May 25 23:53:55.602022 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:55.601910    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"
May 25 23:53:55.604286 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:55.604263    2578 generic.go:358] "Generic (PLEG): container finished" podID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerID="d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce" exitCode=0
May 25 23:53:55.604439 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:55.604321    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerDied","Data":"d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce"}
May 25 23:53:55.618878 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:55.618848    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"]
May 25 23:53:55.622329 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:55.622302    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6bfd8475f9-z7vt7"]
May 25 23:53:57.007877 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.007842    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80473cc5-d469-4fdb-a1a4-8e04be8e7c8a" path="/var/lib/kubelet/pods/80473cc5-d469-4fdb-a1a4-8e04be8e7c8a/volumes"
May 25 23:53:57.032171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.032134    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"]
May 25 23:53:57.032357 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.032336    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7" podUID="5d629baa-d80b-42ad-83ca-f36d2f444297" containerName="postgres" containerID="cri-o://f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c" gracePeriod=30
May 25 23:53:57.036583 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.036557    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-plkg2"]
May 25 23:53:57.036890 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.036832    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-plkg2" podUID="8ea94386-527a-4d99-acfa-18dd22657443" containerName="redis" containerID="cri-o://836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425" gracePeriod=30
May 25 23:53:57.300168 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.300147    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:53:57.303259 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.303234    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-plkg2"
May 25 23:53:57.360958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.360921    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5d629baa-d80b-42ad-83ca-f36d2f444297-postgresdata\") pod \"5d629baa-d80b-42ad-83ca-f36d2f444297\" (UID: \"5d629baa-d80b-42ad-83ca-f36d2f444297\") "
May 25 23:53:57.361154 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.360970    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6cnm\" (UniqueName: \"kubernetes.io/projected/5d629baa-d80b-42ad-83ca-f36d2f444297-kube-api-access-s6cnm\") pod \"5d629baa-d80b-42ad-83ca-f36d2f444297\" (UID: \"5d629baa-d80b-42ad-83ca-f36d2f444297\") "
May 25 23:53:57.361154 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.360995    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8tpl\" (UniqueName: \"kubernetes.io/projected/8ea94386-527a-4d99-acfa-18dd22657443-kube-api-access-k8tpl\") pod \"8ea94386-527a-4d99-acfa-18dd22657443\" (UID: \"8ea94386-527a-4d99-acfa-18dd22657443\") "
May 25 23:53:57.363310 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.363266    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d629baa-d80b-42ad-83ca-f36d2f444297-kube-api-access-s6cnm" (OuterVolumeSpecName: "kube-api-access-s6cnm") pod "5d629baa-d80b-42ad-83ca-f36d2f444297" (UID: "5d629baa-d80b-42ad-83ca-f36d2f444297"). InnerVolumeSpecName "kube-api-access-s6cnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:53:57.363443 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.363309    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea94386-527a-4d99-acfa-18dd22657443-kube-api-access-k8tpl" (OuterVolumeSpecName: "kube-api-access-k8tpl") pod "8ea94386-527a-4d99-acfa-18dd22657443" (UID: "8ea94386-527a-4d99-acfa-18dd22657443"). InnerVolumeSpecName "kube-api-access-k8tpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:53:57.392117 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.392063    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d629baa-d80b-42ad-83ca-f36d2f444297-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "5d629baa-d80b-42ad-83ca-f36d2f444297" (UID: "5d629baa-d80b-42ad-83ca-f36d2f444297"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:53:57.461983 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.461949    2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5d629baa-d80b-42ad-83ca-f36d2f444297-postgresdata\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:53:57.461983 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.461982    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6cnm\" (UniqueName: \"kubernetes.io/projected/5d629baa-d80b-42ad-83ca-f36d2f444297-kube-api-access-s6cnm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:53:57.462139 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.461993    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8tpl\" (UniqueName: \"kubernetes.io/projected/8ea94386-527a-4d99-acfa-18dd22657443-kube-api-access-k8tpl\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:53:57.611910 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.611876    2578 generic.go:358] "Generic (PLEG): container finished" podID="5d629baa-d80b-42ad-83ca-f36d2f444297" containerID="f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c" exitCode=0
May 25 23:53:57.612066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.611935    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"
May 25 23:53:57.612066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.611958    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7" event={"ID":"5d629baa-d80b-42ad-83ca-f36d2f444297","Type":"ContainerDied","Data":"f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c"}
May 25 23:53:57.612066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.611994    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-zmzq7" event={"ID":"5d629baa-d80b-42ad-83ca-f36d2f444297","Type":"ContainerDied","Data":"3b08e1278b6cd5faedda1914b24fe591d25fe392a2dc9c6997f5521e442e037a"}
May 25 23:53:57.612066 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.612012    2578 scope.go:117] "RemoveContainer" containerID="f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c"
May 25 23:53:57.613221 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.613200    2578 generic.go:358] "Generic (PLEG): container finished" podID="8ea94386-527a-4d99-acfa-18dd22657443" containerID="836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425" exitCode=0
May 25 23:53:57.613328 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.613230    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-plkg2" event={"ID":"8ea94386-527a-4d99-acfa-18dd22657443","Type":"ContainerDied","Data":"836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425"}
May 25 23:53:57.613328 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.613253    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-plkg2" event={"ID":"8ea94386-527a-4d99-acfa-18dd22657443","Type":"ContainerDied","Data":"d1c0c1cdf8ab438493999c28550385eb4c11e610b0f367855d78935ab8cf65d2"}
May 25 23:53:57.613328 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.613270    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-plkg2"
May 25 23:53:57.621416 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.621398    2578 scope.go:117] "RemoveContainer" containerID="f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c"
May 25 23:53:57.621740 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:53:57.621700    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c\": container with ID starting with f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c not found: ID does not exist" containerID="f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c"
May 25 23:53:57.621849 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.621757    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c"} err="failed to get container status \"f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c\": rpc error: code = NotFound desc = could not find container \"f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c\": container with ID starting with f24b558879447d722adebbbcc9bdd5a5fae61a85153ebabe8e2fbba93d1daf7c not found: ID does not exist"
May 25 23:53:57.621849 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.621784    2578 scope.go:117] "RemoveContainer" containerID="836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425"
May 25 23:53:57.629288 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.629269    2578 scope.go:117] "RemoveContainer" containerID="836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425"
May 25 23:53:57.629558 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:53:57.629536    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425\": container with ID starting with 836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425 not found: ID does not exist" containerID="836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425"
May 25 23:53:57.629614 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.629569    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425"} err="failed to get container status \"836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425\": rpc error: code = NotFound desc = could not find container \"836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425\": container with ID starting with 836632ee9e8a58bb5d7b12868a97ebbfa2b8a9348148ecb3781fb3e9e5e0e425 not found: ID does not exist"
May 25 23:53:57.636438 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.636411    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-plkg2"]
May 25 23:53:57.639745 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.639706    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-plkg2"]
May 25 23:53:57.651249 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.651218    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"]
May 25 23:53:57.654737 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:57.654696    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-zmzq7"]
May 25 23:53:59.007180 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:59.007143    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d629baa-d80b-42ad-83ca-f36d2f444297" path="/var/lib/kubelet/pods/5d629baa-d80b-42ad-83ca-f36d2f444297/volumes"
May 25 23:53:59.007551 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:53:59.007454    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea94386-527a-4d99-acfa-18dd22657443" path="/var/lib/kubelet/pods/8ea94386-527a-4d99-acfa-18dd22657443/volumes"
May 25 23:54:00.726588 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:00.726547    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 25 23:54:02.283545 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:02.283497    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 25 23:54:05.997698 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.997658    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"]
May 25 23:54:05.998210 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.998111    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80473cc5-d469-4fdb-a1a4-8e04be8e7c8a" containerName="manager"
May 25 23:54:05.998210 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.998131    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80473cc5-d469-4fdb-a1a4-8e04be8e7c8a" containerName="manager"
May 25 23:54:05.998210 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.998162    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea94386-527a-4d99-acfa-18dd22657443" containerName="redis"
May 25 23:54:05.998210 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.998170    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea94386-527a-4d99-acfa-18dd22657443" containerName="redis"
May 25 23:54:05.998210 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.998187    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d629baa-d80b-42ad-83ca-f36d2f444297" containerName="postgres"
May 25 23:54:05.998210 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.998195    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d629baa-d80b-42ad-83ca-f36d2f444297" containerName="postgres"
May 25 23:54:05.998509 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.998261    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="80473cc5-d469-4fdb-a1a4-8e04be8e7c8a" containerName="manager"
May 25 23:54:05.998509 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.998273    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="8ea94386-527a-4d99-acfa-18dd22657443" containerName="redis"
May 25 23:54:05.998509 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:05.998287    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5d629baa-d80b-42ad-83ca-f36d2f444297" containerName="postgres"
May 25 23:54:06.006194 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.006153    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"
May 25 23:54:06.010344 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.010176    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-49l9g\""
May 25 23:54:06.010344 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.010183    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 25 23:54:06.010344 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.010265    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 25 23:54:06.010654 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.010630    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"]
May 25 23:54:06.037214 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.037176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf45m\" (UniqueName: \"kubernetes.io/projected/58618b25-d000-44f2-9cac-92acc6ed4816-kube-api-access-kf45m\") pod \"feast-operator-controller-manager-cf99db75-b258b\" (UID: \"58618b25-d000-44f2-9cac-92acc6ed4816\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"
May 25 23:54:06.138141 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.138102    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kf45m\" (UniqueName: \"kubernetes.io/projected/58618b25-d000-44f2-9cac-92acc6ed4816-kube-api-access-kf45m\") pod \"feast-operator-controller-manager-cf99db75-b258b\" (UID: \"58618b25-d000-44f2-9cac-92acc6ed4816\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"
May 25 23:54:06.150223 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.150183    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf45m\" (UniqueName: \"kubernetes.io/projected/58618b25-d000-44f2-9cac-92acc6ed4816-kube-api-access-kf45m\") pod \"feast-operator-controller-manager-cf99db75-b258b\" (UID: \"58618b25-d000-44f2-9cac-92acc6ed4816\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"
May 25 23:54:06.316715 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.316673    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"
May 25 23:54:06.453524 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.453481    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"]
May 25 23:54:06.456547 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:54:06.456518    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58618b25_d000_44f2_9cac_92acc6ed4816.slice/crio-c5516a42488acc745a93f67b9e005bc4adedc3f4541cb6e3ac12115d85d804e9 WatchSource:0}: Error finding container c5516a42488acc745a93f67b9e005bc4adedc3f4541cb6e3ac12115d85d804e9: Status 404 returned error can't find the container with id c5516a42488acc745a93f67b9e005bc4adedc3f4541cb6e3ac12115d85d804e9
May 25 23:54:06.458607 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.458588    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 23:54:06.641819 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:06.641706    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b" event={"ID":"58618b25-d000-44f2-9cac-92acc6ed4816","Type":"ContainerStarted","Data":"c5516a42488acc745a93f67b9e005bc4adedc3f4541cb6e3ac12115d85d804e9"}
May 25 23:54:08.648289 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:08.648251    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b" event={"ID":"58618b25-d000-44f2-9cac-92acc6ed4816","Type":"ContainerStarted","Data":"75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509"}
May 25 23:54:08.648777 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:08.648499    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"
May 25 23:54:08.666876 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:08.666811    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b" podStartSLOduration=1.919959792 podStartE2EDuration="3.666795143s" podCreationTimestamp="2026-05-25 23:54:05 +0000 UTC" firstStartedPulling="2026-05-25 23:54:06.458775468 +0000 UTC m=+1146.013719707" lastFinishedPulling="2026-05-25 23:54:08.205610803 +0000 UTC m=+1147.760555058" observedRunningTime="2026-05-25 23:54:08.665868252 +0000 UTC m=+1148.220812515" watchObservedRunningTime="2026-05-25 23:54:08.666795143 +0000 UTC m=+1148.221739447"
May 25 23:54:10.726992 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:10.726938    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 25 23:54:12.284163 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:12.284123    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 25 23:54:19.653241 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:19.653157    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"
May 25 23:54:20.726803 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:20.726759    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 25 23:54:20.727261 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:20.726892    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:54:22.283646 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:22.283604    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 25 23:54:22.284037 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:22.283757    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:54:25.313002 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.312977    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:54:25.334776 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.334750    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:54:25.406196 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.406157    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-online-tls\") pod \"522a6a19-f6de-4468-8c15-69881db6961f\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") "
May 25 23:54:25.406196 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.406198    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-registry-tls\") pod \"f64e3865-c764-4d7f-b577-3fa70762b31e\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") "
May 25 23:54:25.406423 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.406220    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/522a6a19-f6de-4468-8c15-69881db6961f-feast-data\") pod \"522a6a19-f6de-4468-8c15-69881db6961f\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") "
May 25 23:54:25.406423 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.406271    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg9zk\" (UniqueName: \"kubernetes.io/projected/522a6a19-f6de-4468-8c15-69881db6961f-kube-api-access-cg9zk\") pod \"522a6a19-f6de-4468-8c15-69881db6961f\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") "
May 25 23:54:25.406423 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.406291    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-online-tls\") pod \"f64e3865-c764-4d7f-b577-3fa70762b31e\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") "
May 25 23:54:25.406423 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.406311    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xhp2\" (UniqueName: \"kubernetes.io/projected/f64e3865-c764-4d7f-b577-3fa70762b31e-kube-api-access-5xhp2\") pod \"f64e3865-c764-4d7f-b577-3fa70762b31e\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") "
May 25 23:54:25.406423 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.406351    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-registry-tls\") pod \"522a6a19-f6de-4468-8c15-69881db6961f\" (UID: \"522a6a19-f6de-4468-8c15-69881db6961f\") "
May 25 23:54:25.406423 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.406371    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f64e3865-c764-4d7f-b577-3fa70762b31e-feast-data\") pod \"f64e3865-c764-4d7f-b577-3fa70762b31e\" (UID: \"f64e3865-c764-4d7f-b577-3fa70762b31e\") "
May 25 23:54:25.407136 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.407052    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64e3865-c764-4d7f-b577-3fa70762b31e-feast-data" (OuterVolumeSpecName: "feast-data") pod "f64e3865-c764-4d7f-b577-3fa70762b31e" (UID: "f64e3865-c764-4d7f-b577-3fa70762b31e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:54:25.408958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.408885    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f64e3865-c764-4d7f-b577-3fa70762b31e" (UID: "f64e3865-c764-4d7f-b577-3fa70762b31e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:54:25.409069 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.408994    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "522a6a19-f6de-4468-8c15-69881db6961f" (UID: "522a6a19-f6de-4468-8c15-69881db6961f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:54:25.409149 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.409092    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-online-tls" (OuterVolumeSpecName: "online-tls") pod "f64e3865-c764-4d7f-b577-3fa70762b31e" (UID: "f64e3865-c764-4d7f-b577-3fa70762b31e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:54:25.409149 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.409109    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-online-tls" (OuterVolumeSpecName: "online-tls") pod "522a6a19-f6de-4468-8c15-69881db6961f" (UID: "522a6a19-f6de-4468-8c15-69881db6961f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 23:54:25.409149 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.409116    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64e3865-c764-4d7f-b577-3fa70762b31e-kube-api-access-5xhp2" (OuterVolumeSpecName: "kube-api-access-5xhp2") pod "f64e3865-c764-4d7f-b577-3fa70762b31e" (UID: "f64e3865-c764-4d7f-b577-3fa70762b31e"). InnerVolumeSpecName "kube-api-access-5xhp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:54:25.409149 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.409113    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522a6a19-f6de-4468-8c15-69881db6961f-kube-api-access-cg9zk" (OuterVolumeSpecName: "kube-api-access-cg9zk") pod "522a6a19-f6de-4468-8c15-69881db6961f" (UID: "522a6a19-f6de-4468-8c15-69881db6961f"). InnerVolumeSpecName "kube-api-access-cg9zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:54:25.415963 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.415932    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522a6a19-f6de-4468-8c15-69881db6961f-feast-data" (OuterVolumeSpecName: "feast-data") pod "522a6a19-f6de-4468-8c15-69881db6961f" (UID: "522a6a19-f6de-4468-8c15-69881db6961f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 23:54:25.507171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.507083    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg9zk\" (UniqueName: \"kubernetes.io/projected/522a6a19-f6de-4468-8c15-69881db6961f-kube-api-access-cg9zk\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:54:25.507171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.507118    2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-online-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:54:25.507171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.507128    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhp2\" (UniqueName: \"kubernetes.io/projected/f64e3865-c764-4d7f-b577-3fa70762b31e-kube-api-access-5xhp2\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:54:25.507171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.507140    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-registry-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:54:25.507171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.507150    2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f64e3865-c764-4d7f-b577-3fa70762b31e-feast-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:54:25.507171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.507159    2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/522a6a19-f6de-4468-8c15-69881db6961f-online-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:54:25.507171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.507167    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f64e3865-c764-4d7f-b577-3fa70762b31e-registry-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:54:25.507171 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.507175    2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/522a6a19-f6de-4468-8c15-69881db6961f-feast-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:54:25.704690 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.704648    2578 generic.go:358] "Generic (PLEG): container finished" podID="522a6a19-f6de-4468-8c15-69881db6961f" containerID="d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624" exitCode=137
May 25 23:54:25.704889 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.704754    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"
May 25 23:54:25.704889 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.704752    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerDied","Data":"d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624"}
May 25 23:54:25.704889 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.704795    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz" event={"ID":"522a6a19-f6de-4468-8c15-69881db6961f","Type":"ContainerDied","Data":"00f0ca9ebabbd3ec4d85f1a3c0ee24c7f203e5aa28324ab1d0866a529710146e"}
May 25 23:54:25.704889 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.704819    2578 scope.go:117] "RemoveContainer" containerID="c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e"
May 25 23:54:25.706797 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.706774    2578 generic.go:358] "Generic (PLEG): container finished" podID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerID="c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c" exitCode=137
May 25 23:54:25.706932 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.706855    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerDied","Data":"c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c"}
May 25 23:54:25.706932 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.706884    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv" event={"ID":"f64e3865-c764-4d7f-b577-3fa70762b31e","Type":"ContainerDied","Data":"b3904b2423d8b4c6a8a40bb532870b3017cdcf738579c6c5e0f38205f26d2c71"}
May 25 23:54:25.706932 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.706912    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"
May 25 23:54:25.714557 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.714537    2578 scope.go:117] "RemoveContainer" containerID="d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624"
May 25 23:54:25.724453 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.724431    2578 scope.go:117] "RemoveContainer" containerID="dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4"
May 25 23:54:25.727130 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.727104    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"]
May 25 23:54:25.733680 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.733147    2578 scope.go:117] "RemoveContainer" containerID="bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3"
May 25 23:54:25.733680 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.733243    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-984d96475-2t4gz"]
May 25 23:54:25.740932 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.740913    2578 scope.go:117] "RemoveContainer" containerID="c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e"
May 25 23:54:25.741217 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:54:25.741195    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e\": container with ID starting with c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e not found: ID does not exist" containerID="c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e"
May 25 23:54:25.741277 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.741226    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e"} err="failed to get container status \"c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e\": rpc error: code = NotFound desc = could not find container \"c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e\": container with ID starting with c76e76097b71ce3692c2e9f5fcc0b96aa41305ad4df4ee552a8bd41ca6fc815e not found: ID does not exist"
May 25 23:54:25.741277 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.741247    2578 scope.go:117] "RemoveContainer" containerID="d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624"
May 25 23:54:25.741489 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:54:25.741472    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624\": container with ID starting with d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624 not found: ID does not exist" containerID="d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624"
May 25 23:54:25.741555 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.741498    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624"} err="failed to get container status \"d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624\": rpc error: code = NotFound desc = could not find container \"d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624\": container with ID starting with d1426f235f7289302a258df07c005f3ecf860ec732e4739449f9fdf43d741624 not found: ID does not exist"
May 25 23:54:25.741555 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.741520    2578 scope.go:117] "RemoveContainer" containerID="dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4"
May 25 23:54:25.741865 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:54:25.741842    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4\": container with ID starting with dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4 not found: ID does not exist" containerID="dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4"
May 25 23:54:25.741978 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.741868    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4"} err="failed to get container status \"dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4\": rpc error: code = NotFound desc = could not find container \"dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4\": container with ID starting with dee6b2fa22d48c8572afc95bc616f3075e74fcab33c0d85ae78d0a9351be56b4 not found: ID does not exist"
May 25 23:54:25.741978 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.741884    2578 scope.go:117] "RemoveContainer" containerID="bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3"
May 25 23:54:25.742515 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:54:25.742175    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3\": container with ID starting with bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3 not found: ID does not exist" containerID="bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3"
May 25 23:54:25.742515 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.742208    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3"} err="failed to get container status \"bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3\": rpc error: code = NotFound desc = could not find container \"bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3\": container with ID starting with bf03907dc44a03f9c36b13540528a016c1d911318c5b1f0aa5cabe25309359c3 not found: ID does not exist"
May 25 23:54:25.742515 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.742229    2578 scope.go:117] "RemoveContainer" containerID="d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce"
May 25 23:54:25.743916 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.743896    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"]
May 25 23:54:25.749061 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.749033    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8dcdfbc44-knswv"]
May 25 23:54:25.750445 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.750432    2578 scope.go:117] "RemoveContainer" containerID="c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c"
May 25 23:54:25.759300 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.759278    2578 scope.go:117] "RemoveContainer" containerID="6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6"
May 25 23:54:25.766828 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.766796    2578 scope.go:117] "RemoveContainer" containerID="d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224"
May 25 23:54:25.773924 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.773905    2578 scope.go:117] "RemoveContainer" containerID="d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce"
May 25 23:54:25.774201 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:54:25.774185    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce\": container with ID starting with d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce not found: ID does not exist" containerID="d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce"
May 25 23:54:25.774238 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.774209    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce"} err="failed to get container status \"d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce\": rpc error: code = NotFound desc = could not find container \"d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce\": container with ID starting with d981bdd19a7c78dc915b0d6c5a22683189dace76f1d65ce3ea104bd21631a6ce not found: ID does not exist"
May 25 23:54:25.774238 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.774228    2578 scope.go:117] "RemoveContainer" containerID="c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c"
May 25 23:54:25.774442 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:54:25.774425    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c\": container with ID starting with c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c not found: ID does not exist" containerID="c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c"
May 25 23:54:25.774493 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.774444    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c"} err="failed to get container status \"c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c\": rpc error: code = NotFound desc = could not find container \"c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c\": container with ID starting with c4afa48d372afcd297f4e4106c4638a16c87fb62ee2861716ece84797586ba5c not found: ID does not exist"
May 25 23:54:25.774493 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.774458    2578 scope.go:117] "RemoveContainer" containerID="6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6"
May 25 23:54:25.774672 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:54:25.774652    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6\": container with ID starting with 6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6 not found: ID does not exist" containerID="6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6"
May 25 23:54:25.774711 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.774677    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6"} err="failed to get container status \"6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6\": rpc error: code = NotFound desc = could not find container \"6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6\": container with ID starting with 6aa4e958f865515cacdbc13ac1c0c936cadd1900c7e65b1b0daa46b5d8f5d7f6 not found: ID does not exist"
May 25 23:54:25.774711 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.774695    2578 scope.go:117] "RemoveContainer" containerID="d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224"
May 25 23:54:25.774968 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:54:25.774953    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224\": container with ID starting with d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224 not found: ID does not exist" containerID="d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224"
May 25 23:54:25.775010 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:25.774975    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224"} err="failed to get container status \"d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224\": rpc error: code = NotFound desc = could not find container \"d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224\": container with ID starting with d6449887676c7c8a6e935c8d58a9505d8f8b30b30521f9b65b29334924444224 not found: ID does not exist"
May 25 23:54:27.007459 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:27.007423    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522a6a19-f6de-4468-8c15-69881db6961f" path="/var/lib/kubelet/pods/522a6a19-f6de-4468-8c15-69881db6961f/volumes"
May 25 23:54:27.007957 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:54:27.007943    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" path="/var/lib/kubelet/pods/f64e3865-c764-4d7f-b577-3fa70762b31e/volumes"
May 25 23:55:00.989947 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:55:00.989914    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:55:00.991540 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:55:00.991519    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 25 23:59:26.294958 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.294920    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"]
May 25 23:59:26.295524 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.295172    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b" podUID="58618b25-d000-44f2-9cac-92acc6ed4816" containerName="manager" containerID="cri-o://75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509" gracePeriod=10
May 25 23:59:26.534937 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.534914    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"
May 25 23:59:26.563744 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.563688    2578 generic.go:358] "Generic (PLEG): container finished" podID="58618b25-d000-44f2-9cac-92acc6ed4816" containerID="75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509" exitCode=0
May 25 23:59:26.563953 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.563759    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b" event={"ID":"58618b25-d000-44f2-9cac-92acc6ed4816","Type":"ContainerDied","Data":"75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509"}
May 25 23:59:26.563953 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.563797    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b" event={"ID":"58618b25-d000-44f2-9cac-92acc6ed4816","Type":"ContainerDied","Data":"c5516a42488acc745a93f67b9e005bc4adedc3f4541cb6e3ac12115d85d804e9"}
May 25 23:59:26.563953 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.563817    2578 scope.go:117] "RemoveContainer" containerID="75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509"
May 25 23:59:26.563953 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.563815    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"
May 25 23:59:26.572932 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.572909    2578 scope.go:117] "RemoveContainer" containerID="75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509"
May 25 23:59:26.573257 ip-10-0-130-28 kubenswrapper[2578]: E0525 23:59:26.573232    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509\": container with ID starting with 75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509 not found: ID does not exist" containerID="75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509"
May 25 23:59:26.573337 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.573271    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509"} err="failed to get container status \"75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509\": rpc error: code = NotFound desc = could not find container \"75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509\": container with ID starting with 75744dc64fd67f32273199b8471006df2286bb92af26cf1f718b0cbc61d59509 not found: ID does not exist"
May 25 23:59:26.680333 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.680292    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf45m\" (UniqueName: \"kubernetes.io/projected/58618b25-d000-44f2-9cac-92acc6ed4816-kube-api-access-kf45m\") pod \"58618b25-d000-44f2-9cac-92acc6ed4816\" (UID: \"58618b25-d000-44f2-9cac-92acc6ed4816\") "
May 25 23:59:26.682487 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.682458    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58618b25-d000-44f2-9cac-92acc6ed4816-kube-api-access-kf45m" (OuterVolumeSpecName: "kube-api-access-kf45m") pod "58618b25-d000-44f2-9cac-92acc6ed4816" (UID: "58618b25-d000-44f2-9cac-92acc6ed4816"). InnerVolumeSpecName "kube-api-access-kf45m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 23:59:26.781946 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.781902    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf45m\" (UniqueName: \"kubernetes.io/projected/58618b25-d000-44f2-9cac-92acc6ed4816-kube-api-access-kf45m\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 25 23:59:26.886311 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.886276    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"]
May 25 23:59:26.890085 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:26.890058    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-b258b"]
May 25 23:59:27.007470 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:27.007439    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58618b25-d000-44f2-9cac-92acc6ed4816" path="/var/lib/kubelet/pods/58618b25-d000-44f2-9cac-92acc6ed4816/volumes"
May 25 23:59:30.646940 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.646906    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"]
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647177    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="feast-init"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647187    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="feast-init"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647201    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="online"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647207    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="online"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647215    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="registry"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647220    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="registry"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647228    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="registry"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647233    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="registry"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647240    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58618b25-d000-44f2-9cac-92acc6ed4816" containerName="manager"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647245    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58618b25-d000-44f2-9cac-92acc6ed4816" containerName="manager"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647252    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="feast-apply"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647257    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="feast-apply"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647266    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="feast-init"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647271    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="feast-init"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647277    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="feast-apply"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647283    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="feast-apply"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647288    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="online"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647293    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="online"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647337    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="58618b25-d000-44f2-9cac-92acc6ed4816" containerName="manager"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647346    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="registry"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647353    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="522a6a19-f6de-4468-8c15-69881db6961f" containerName="online"
May 25 23:59:30.647353 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647360    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="online"
May 25 23:59:30.648010 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.647367    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f64e3865-c764-4d7f-b577-3fa70762b31e" containerName="registry"
May 25 23:59:30.649370 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.649352    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"
May 25 23:59:30.651639 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.651617    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 25 23:59:30.651831 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.651639    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-49l9g\""
May 25 23:59:30.652642 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.652623    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 25 23:59:30.659939 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.659912    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"]
May 25 23:59:30.711987 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.711942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6nr5\" (UniqueName: \"kubernetes.io/projected/32ca290f-b725-4711-8514-e01f4db0d76c-kube-api-access-k6nr5\") pod \"feast-operator-controller-manager-cf99db75-8qcjl\" (UID: \"32ca290f-b725-4711-8514-e01f4db0d76c\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"
May 25 23:59:30.813120 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.813087    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k6nr5\" (UniqueName: \"kubernetes.io/projected/32ca290f-b725-4711-8514-e01f4db0d76c-kube-api-access-k6nr5\") pod \"feast-operator-controller-manager-cf99db75-8qcjl\" (UID: \"32ca290f-b725-4711-8514-e01f4db0d76c\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"
May 25 23:59:30.823157 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.823123    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6nr5\" (UniqueName: \"kubernetes.io/projected/32ca290f-b725-4711-8514-e01f4db0d76c-kube-api-access-k6nr5\") pod \"feast-operator-controller-manager-cf99db75-8qcjl\" (UID: \"32ca290f-b725-4711-8514-e01f4db0d76c\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"
May 25 23:59:30.959891 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:30.959801    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"
May 25 23:59:31.086290 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:31.086251    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"]
May 25 23:59:31.089059 ip-10-0-130-28 kubenswrapper[2578]: W0525 23:59:31.089027    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ca290f_b725_4711_8514_e01f4db0d76c.slice/crio-28944c443e12928866c456c147a926b562ac31e707cdd8a833bad2e3ad9cf5e4 WatchSource:0}: Error finding container 28944c443e12928866c456c147a926b562ac31e707cdd8a833bad2e3ad9cf5e4: Status 404 returned error can't find the container with id 28944c443e12928866c456c147a926b562ac31e707cdd8a833bad2e3ad9cf5e4
May 25 23:59:31.090852 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:31.090835    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 23:59:31.579546 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:31.579508    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl" event={"ID":"32ca290f-b725-4711-8514-e01f4db0d76c","Type":"ContainerStarted","Data":"b93d03da59f415eb648fdf71a1c5eb366c91829f1c99ec572419fd970d7102d5"}
May 25 23:59:31.579546 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:31.579544    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl" event={"ID":"32ca290f-b725-4711-8514-e01f4db0d76c","Type":"ContainerStarted","Data":"28944c443e12928866c456c147a926b562ac31e707cdd8a833bad2e3ad9cf5e4"}
May 25 23:59:31.579845 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:31.579678    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"
May 25 23:59:31.596921 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:31.596878    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl" podStartSLOduration=1.596864115 podStartE2EDuration="1.596864115s" podCreationTimestamp="2026-05-25 23:59:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 23:59:31.595699642 +0000 UTC m=+1471.150643906" watchObservedRunningTime="2026-05-25 23:59:31.596864115 +0000 UTC m=+1471.151808376"
May 25 23:59:42.584478 ip-10-0-130-28 kubenswrapper[2578]: I0525 23:59:42.584444    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-8qcjl"
May 26 00:00:00.136015 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.135980    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29662560-v8wdt"]
May 26 00:00:00.138057 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.138041    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:00:00.141518 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.141455    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
May 26 00:00:00.142430 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.142410    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-jwshv\""
May 26 00:00:00.151905 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.151873    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29662560-v8wdt"]
May 26 00:00:00.239629 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.239596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dbada13-a66e-4b9a-8f4c-8be6289650b7-serviceca\") pod \"image-pruner-29662560-v8wdt\" (UID: \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\") " pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:00:00.239629 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.239635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfkp2\" (UniqueName: \"kubernetes.io/projected/7dbada13-a66e-4b9a-8f4c-8be6289650b7-kube-api-access-nfkp2\") pod \"image-pruner-29662560-v8wdt\" (UID: \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\") " pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:00:00.340081 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.340041    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dbada13-a66e-4b9a-8f4c-8be6289650b7-serviceca\") pod \"image-pruner-29662560-v8wdt\" (UID: \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\") " pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:00:00.340225 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.340091    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nfkp2\" (UniqueName: \"kubernetes.io/projected/7dbada13-a66e-4b9a-8f4c-8be6289650b7-kube-api-access-nfkp2\") pod \"image-pruner-29662560-v8wdt\" (UID: \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\") " pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:00:00.340621 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.340598    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dbada13-a66e-4b9a-8f4c-8be6289650b7-serviceca\") pod \"image-pruner-29662560-v8wdt\" (UID: \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\") " pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:00:00.348973 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.348940    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfkp2\" (UniqueName: \"kubernetes.io/projected/7dbada13-a66e-4b9a-8f4c-8be6289650b7-kube-api-access-nfkp2\") pod \"image-pruner-29662560-v8wdt\" (UID: \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\") " pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:00:00.469883 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.469843    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:00:00.598568 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.598479    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29662560-v8wdt"]
May 26 00:00:00.601571 ip-10-0-130-28 kubenswrapper[2578]: W0526 00:00:00.601541    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbada13_a66e_4b9a_8f4c_8be6289650b7.slice/crio-da8c78a8c147122fd191f171466e1660bd1cf3750b86e8df36f74ae3b3ae2755 WatchSource:0}: Error finding container da8c78a8c147122fd191f171466e1660bd1cf3750b86e8df36f74ae3b3ae2755: Status 404 returned error can't find the container with id da8c78a8c147122fd191f171466e1660bd1cf3750b86e8df36f74ae3b3ae2755
May 26 00:00:00.659450 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:00.659424    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29662560-v8wdt" event={"ID":"7dbada13-a66e-4b9a-8f4c-8be6289650b7","Type":"ContainerStarted","Data":"da8c78a8c147122fd191f171466e1660bd1cf3750b86e8df36f74ae3b3ae2755"}
May 26 00:00:01.016301 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:01.016272    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 26 00:00:01.017189 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:01.017165    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 26 00:00:01.664064 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:01.663967    2578 generic.go:358] "Generic (PLEG): container finished" podID="7dbada13-a66e-4b9a-8f4c-8be6289650b7" containerID="ab99e1501baff2fd82f5e263e8af6f0702cbdec1abec9fe3493e0df791b3501a" exitCode=0
May 26 00:00:01.664064 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:01.664030    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29662560-v8wdt" event={"ID":"7dbada13-a66e-4b9a-8f4c-8be6289650b7","Type":"ContainerDied","Data":"ab99e1501baff2fd82f5e263e8af6f0702cbdec1abec9fe3493e0df791b3501a"}
May 26 00:00:02.788948 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:02.788921    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:00:02.860502 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:02.860470    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dbada13-a66e-4b9a-8f4c-8be6289650b7-serviceca\") pod \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\" (UID: \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\") "
May 26 00:00:02.860674 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:02.860530    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfkp2\" (UniqueName: \"kubernetes.io/projected/7dbada13-a66e-4b9a-8f4c-8be6289650b7-kube-api-access-nfkp2\") pod \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\" (UID: \"7dbada13-a66e-4b9a-8f4c-8be6289650b7\") "
May 26 00:00:02.860880 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:02.860854    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbada13-a66e-4b9a-8f4c-8be6289650b7-serviceca" (OuterVolumeSpecName: "serviceca") pod "7dbada13-a66e-4b9a-8f4c-8be6289650b7" (UID: "7dbada13-a66e-4b9a-8f4c-8be6289650b7"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 00:00:02.862789 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:02.862761    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbada13-a66e-4b9a-8f4c-8be6289650b7-kube-api-access-nfkp2" (OuterVolumeSpecName: "kube-api-access-nfkp2") pod "7dbada13-a66e-4b9a-8f4c-8be6289650b7" (UID: "7dbada13-a66e-4b9a-8f4c-8be6289650b7"). InnerVolumeSpecName "kube-api-access-nfkp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 00:00:02.961289 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:02.961183    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfkp2\" (UniqueName: \"kubernetes.io/projected/7dbada13-a66e-4b9a-8f4c-8be6289650b7-kube-api-access-nfkp2\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 26 00:00:02.961289 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:02.961229    2578 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dbada13-a66e-4b9a-8f4c-8be6289650b7-serviceca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 26 00:00:03.670775 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:03.670740    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29662560-v8wdt" event={"ID":"7dbada13-a66e-4b9a-8f4c-8be6289650b7","Type":"ContainerDied","Data":"da8c78a8c147122fd191f171466e1660bd1cf3750b86e8df36f74ae3b3ae2755"}
May 26 00:00:03.670775 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:03.670775    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da8c78a8c147122fd191f171466e1660bd1cf3750b86e8df36f74ae3b3ae2755"
May 26 00:00:03.671010 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:00:03.670783    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29662560-v8wdt"
May 26 00:04:40.725277 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.725240    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q6vc9/must-gather-8mq9p"]
May 26 00:04:40.725805 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.725539    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbada13-a66e-4b9a-8f4c-8be6289650b7" containerName="image-pruner"
May 26 00:04:40.725805 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.725552    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbada13-a66e-4b9a-8f4c-8be6289650b7" containerName="image-pruner"
May 26 00:04:40.725805 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.725613    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7dbada13-a66e-4b9a-8f4c-8be6289650b7" containerName="image-pruner"
May 26 00:04:40.727822 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.727790    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:04:40.730493 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.730467    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6vc9\"/\"openshift-service-ca.crt\""
May 26 00:04:40.730623 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.730534    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q6vc9\"/\"default-dockercfg-t8pw6\""
May 26 00:04:40.731465 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.731450    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6vc9\"/\"kube-root-ca.crt\""
May 26 00:04:40.736304 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.736281    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6vc9/must-gather-8mq9p"]
May 26 00:04:40.822146 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.822107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqqh\" (UniqueName: \"kubernetes.io/projected/8b60e57b-4fb0-4ab9-b25e-513a353cd911-kube-api-access-gxqqh\") pod \"must-gather-8mq9p\" (UID: \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\") " pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:04:40.822332 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.822169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b60e57b-4fb0-4ab9-b25e-513a353cd911-must-gather-output\") pod \"must-gather-8mq9p\" (UID: \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\") " pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:04:40.922758 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.922694    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b60e57b-4fb0-4ab9-b25e-513a353cd911-must-gather-output\") pod \"must-gather-8mq9p\" (UID: \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\") " pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:04:40.922944 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.922801    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqqh\" (UniqueName: \"kubernetes.io/projected/8b60e57b-4fb0-4ab9-b25e-513a353cd911-kube-api-access-gxqqh\") pod \"must-gather-8mq9p\" (UID: \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\") " pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:04:40.923094 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.923067    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b60e57b-4fb0-4ab9-b25e-513a353cd911-must-gather-output\") pod \"must-gather-8mq9p\" (UID: \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\") " pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:04:40.930912 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:40.930880    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqqh\" (UniqueName: \"kubernetes.io/projected/8b60e57b-4fb0-4ab9-b25e-513a353cd911-kube-api-access-gxqqh\") pod \"must-gather-8mq9p\" (UID: \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\") " pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:04:41.052278 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:41.052246    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:04:41.178126 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:41.178100    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6vc9/must-gather-8mq9p"]
May 26 00:04:41.180440 ip-10-0-130-28 kubenswrapper[2578]: W0526 00:04:41.180406    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b60e57b_4fb0_4ab9_b25e_513a353cd911.slice/crio-b46a098470413debf6313e4f8356c3b7217ed76b909112dccd1540ab8432f603 WatchSource:0}: Error finding container b46a098470413debf6313e4f8356c3b7217ed76b909112dccd1540ab8432f603: Status 404 returned error can't find the container with id b46a098470413debf6313e4f8356c3b7217ed76b909112dccd1540ab8432f603
May 26 00:04:41.182201 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:41.182185    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 00:04:41.466379 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:41.466284    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" event={"ID":"8b60e57b-4fb0-4ab9-b25e-513a353cd911","Type":"ContainerStarted","Data":"b46a098470413debf6313e4f8356c3b7217ed76b909112dccd1540ab8432f603"}
May 26 00:04:47.485586 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:47.485546    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" event={"ID":"8b60e57b-4fb0-4ab9-b25e-513a353cd911","Type":"ContainerStarted","Data":"0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520"}
May 26 00:04:48.492404 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:48.492354    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" event={"ID":"8b60e57b-4fb0-4ab9-b25e-513a353cd911","Type":"ContainerStarted","Data":"6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e"}
May 26 00:04:48.510816 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:48.510743    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" podStartSLOduration=2.3969733570000002 podStartE2EDuration="8.510708303s" podCreationTimestamp="2026-05-26 00:04:40 +0000 UTC" firstStartedPulling="2026-05-26 00:04:41.182321587 +0000 UTC m=+1780.737265826" lastFinishedPulling="2026-05-26 00:04:47.296056525 +0000 UTC m=+1786.851000772" observedRunningTime="2026-05-26 00:04:48.509990493 +0000 UTC m=+1788.064934755" watchObservedRunningTime="2026-05-26 00:04:48.510708303 +0000 UTC m=+1788.065652565"
May 26 00:04:57.520673 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:57.520581    2578 generic.go:358] "Generic (PLEG): container finished" podID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" containerID="0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520" exitCode=0
May 26 00:04:57.520673 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:57.520625    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" event={"ID":"8b60e57b-4fb0-4ab9-b25e-513a353cd911","Type":"ContainerDied","Data":"0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520"}
May 26 00:04:57.521121 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:57.520971    2578 scope.go:117] "RemoveContainer" containerID="0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520"
May 26 00:04:58.086283 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.086254    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-q6vc9_must-gather-8mq9p_8b60e57b-4fb0-4ab9-b25e-513a353cd911/gather/0.log"
May 26 00:04:58.550162 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.550136    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8rpn/must-gather-2qnrf"]
May 26 00:04:58.552490 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.552474    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rpn/must-gather-2qnrf"
May 26 00:04:58.554937 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.554915    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8rpn\"/\"kube-root-ca.crt\""
May 26 00:04:58.555836 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.555816    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8rpn\"/\"default-dockercfg-wrltz\""
May 26 00:04:58.555926 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.555818    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8rpn\"/\"openshift-service-ca.crt\""
May 26 00:04:58.561526 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.561502    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rpn/must-gather-2qnrf"]
May 26 00:04:58.684115 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.684064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8-must-gather-output\") pod \"must-gather-2qnrf\" (UID: \"c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8\") " pod="openshift-must-gather-j8rpn/must-gather-2qnrf"
May 26 00:04:58.684289 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.684137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt49x\" (UniqueName: \"kubernetes.io/projected/c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8-kube-api-access-vt49x\") pod \"must-gather-2qnrf\" (UID: \"c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8\") " pod="openshift-must-gather-j8rpn/must-gather-2qnrf"
May 26 00:04:58.785304 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.785268    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8-must-gather-output\") pod \"must-gather-2qnrf\" (UID: \"c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8\") " pod="openshift-must-gather-j8rpn/must-gather-2qnrf"
May 26 00:04:58.785469 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.785325    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vt49x\" (UniqueName: \"kubernetes.io/projected/c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8-kube-api-access-vt49x\") pod \"must-gather-2qnrf\" (UID: \"c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8\") " pod="openshift-must-gather-j8rpn/must-gather-2qnrf"
May 26 00:04:58.785627 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.785607    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8-must-gather-output\") pod \"must-gather-2qnrf\" (UID: \"c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8\") " pod="openshift-must-gather-j8rpn/must-gather-2qnrf"
May 26 00:04:58.792560 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.792531    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt49x\" (UniqueName: \"kubernetes.io/projected/c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8-kube-api-access-vt49x\") pod \"must-gather-2qnrf\" (UID: \"c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8\") " pod="openshift-must-gather-j8rpn/must-gather-2qnrf"
May 26 00:04:58.862536 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.862446    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rpn/must-gather-2qnrf"
May 26 00:04:58.986566 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:58.986526    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rpn/must-gather-2qnrf"]
May 26 00:04:58.989797 ip-10-0-130-28 kubenswrapper[2578]: W0526 00:04:58.989769    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ff8b3c_91b2_4ee0_b9c2_e7712883b1f8.slice/crio-99e341ad3a9f87d2f0872785546d754bfd5a3c189dc539a1db0d9a0b0736f639 WatchSource:0}: Error finding container 99e341ad3a9f87d2f0872785546d754bfd5a3c189dc539a1db0d9a0b0736f639: Status 404 returned error can't find the container with id 99e341ad3a9f87d2f0872785546d754bfd5a3c189dc539a1db0d9a0b0736f639
May 26 00:04:59.526684 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:04:59.526650    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rpn/must-gather-2qnrf" event={"ID":"c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8","Type":"ContainerStarted","Data":"99e341ad3a9f87d2f0872785546d754bfd5a3c189dc539a1db0d9a0b0736f639"}
May 26 00:05:00.531111 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:00.531058    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rpn/must-gather-2qnrf" event={"ID":"c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8","Type":"ContainerStarted","Data":"d9ca9fd958003821c4c838b41885d125c2c21bf2dabeb95318b3a20724fd7575"}
May 26 00:05:00.531610 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:00.531119    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rpn/must-gather-2qnrf" event={"ID":"c2ff8b3c-91b2-4ee0-b9c2-e7712883b1f8","Type":"ContainerStarted","Data":"88327cbd1553674ef36005307e156ffaed524ca8348f6dfa79bd3bd7f9ba3203"}
May 26 00:05:00.546665 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:00.546591    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-j8rpn/must-gather-2qnrf" podStartSLOduration=1.6993682049999999 podStartE2EDuration="2.546567624s" podCreationTimestamp="2026-05-26 00:04:58 +0000 UTC" firstStartedPulling="2026-05-26 00:04:58.991854302 +0000 UTC m=+1798.546798542" lastFinishedPulling="2026-05-26 00:04:59.839053719 +0000 UTC m=+1799.393997961" observedRunningTime="2026-05-26 00:05:00.544952861 +0000 UTC m=+1800.099897124" watchObservedRunningTime="2026-05-26 00:05:00.546567624 +0000 UTC m=+1800.101511886"
May 26 00:05:01.047553 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:01.047144    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 26 00:05:01.049749 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:01.049702    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 26 00:05:01.151840 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:01.151806    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9s245_f0633bd5-0b65-4ce4-9fb5-963307feb3b6/global-pull-secret-syncer/0.log"
May 26 00:05:01.333195 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:01.333158    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g24br_09660ea1-1664-415f-aa07-f0fef147096b/konnectivity-agent/0.log"
May 26 00:05:01.387958 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:01.387927    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-28.ec2.internal_eebff1f1a969cd0c8699443c720bbf73/haproxy/0.log"
May 26 00:05:03.431035 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.430567    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-q6vc9/must-gather-8mq9p"]
May 26 00:05:03.431035 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.430631    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-q6vc9/must-gather-8mq9p"]
May 26 00:05:03.440750 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.433238    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" containerName="copy" containerID="cri-o://6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e" gracePeriod=2
May 26 00:05:03.444397 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.444358    2578 status_manager.go:895] "Failed to get status for pod" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" err="pods \"must-gather-8mq9p\" is forbidden: User \"system:node:ip-10-0-130-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-q6vc9\": no relationship found between node 'ip-10-0-130-28.ec2.internal' and this object"
May 26 00:05:03.823700 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.823421    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-q6vc9_must-gather-8mq9p_8b60e57b-4fb0-4ab9-b25e-513a353cd911/copy/0.log"
May 26 00:05:03.824456 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.824114    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:05:03.826983 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.826937    2578 status_manager.go:895] "Failed to get status for pod" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" err="pods \"must-gather-8mq9p\" is forbidden: User \"system:node:ip-10-0-130-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-q6vc9\": no relationship found between node 'ip-10-0-130-28.ec2.internal' and this object"
May 26 00:05:03.939519 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.938693    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b60e57b-4fb0-4ab9-b25e-513a353cd911-must-gather-output\") pod \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\" (UID: \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\") "
May 26 00:05:03.939519 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.938852    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxqqh\" (UniqueName: \"kubernetes.io/projected/8b60e57b-4fb0-4ab9-b25e-513a353cd911-kube-api-access-gxqqh\") pod \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\" (UID: \"8b60e57b-4fb0-4ab9-b25e-513a353cd911\") "
May 26 00:05:03.940125 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.940069    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b60e57b-4fb0-4ab9-b25e-513a353cd911-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "8b60e57b-4fb0-4ab9-b25e-513a353cd911" (UID: "8b60e57b-4fb0-4ab9-b25e-513a353cd911"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 00:05:03.942958 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:03.942934    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b60e57b-4fb0-4ab9-b25e-513a353cd911-kube-api-access-gxqqh" (OuterVolumeSpecName: "kube-api-access-gxqqh") pod "8b60e57b-4fb0-4ab9-b25e-513a353cd911" (UID: "8b60e57b-4fb0-4ab9-b25e-513a353cd911"). InnerVolumeSpecName "kube-api-access-gxqqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 00:05:04.040214 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.040140    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxqqh\" (UniqueName: \"kubernetes.io/projected/8b60e57b-4fb0-4ab9-b25e-513a353cd911-kube-api-access-gxqqh\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 26 00:05:04.040214 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.040181    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b60e57b-4fb0-4ab9-b25e-513a353cd911-must-gather-output\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
May 26 00:05:04.500071 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.500036    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563d4d67-c7aa-43cc-b6f2-744f2607c29b/alertmanager/0.log"
May 26 00:05:04.524507 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.524388    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563d4d67-c7aa-43cc-b6f2-744f2607c29b/config-reloader/0.log"
May 26 00:05:04.548436 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.548326    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563d4d67-c7aa-43cc-b6f2-744f2607c29b/kube-rbac-proxy-web/0.log"
May 26 00:05:04.550151 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.550074    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-q6vc9_must-gather-8mq9p_8b60e57b-4fb0-4ab9-b25e-513a353cd911/copy/0.log"
May 26 00:05:04.554326 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.550490    2578 generic.go:358] "Generic (PLEG): container finished" podID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" containerID="6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e" exitCode=143
May 26 00:05:04.554326 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.550557    2578 scope.go:117] "RemoveContainer" containerID="6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e"
May 26 00:05:04.554326 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.550565    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6vc9/must-gather-8mq9p"
May 26 00:05:04.554326 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.553119    2578 status_manager.go:895] "Failed to get status for pod" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" err="pods \"must-gather-8mq9p\" is forbidden: User \"system:node:ip-10-0-130-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-q6vc9\": no relationship found between node 'ip-10-0-130-28.ec2.internal' and this object"
May 26 00:05:04.563476 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.563369    2578 scope.go:117] "RemoveContainer" containerID="0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520"
May 26 00:05:04.566851 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.566039    2578 status_manager.go:895] "Failed to get status for pod" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" pod="openshift-must-gather-q6vc9/must-gather-8mq9p" err="pods \"must-gather-8mq9p\" is forbidden: User \"system:node:ip-10-0-130-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-q6vc9\": no relationship found between node 'ip-10-0-130-28.ec2.internal' and this object"
May 26 00:05:04.579741 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.579678    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563d4d67-c7aa-43cc-b6f2-744f2607c29b/kube-rbac-proxy/0.log"
May 26 00:05:04.580378 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.580356    2578 scope.go:117] "RemoveContainer" containerID="6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e"
May 26 00:05:04.580775 ip-10-0-130-28 kubenswrapper[2578]: E0526 00:05:04.580742    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e\": container with ID starting with 6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e not found: ID does not exist" containerID="6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e"
May 26 00:05:04.580871 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.580785    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e"} err="failed to get container status \"6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e\": rpc error: code = NotFound desc = could not find container \"6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e\": container with ID starting with 6373b03f5968a1a64d7e04ec8d2b738b29305212495caa20fddd985e0d24456e not found: ID does not exist"
May 26 00:05:04.580871 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.580813    2578 scope.go:117] "RemoveContainer" containerID="0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520"
May 26 00:05:04.581261 ip-10-0-130-28 kubenswrapper[2578]: E0526 00:05:04.581128    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520\": container with ID starting with 0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520 not found: ID does not exist" containerID="0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520"
May 26 00:05:04.581261 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.581164    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520"} err="failed to get container status \"0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520\": rpc error: code = NotFound desc = could not find container \"0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520\": container with ID starting with 0b8e5a761a36c8ce4a8aa1dbec3c0f8b0d3e0b5320bb67bc3dc3ab70d0241520 not found: ID does not exist"
May 26 00:05:04.603407 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.603369    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563d4d67-c7aa-43cc-b6f2-744f2607c29b/kube-rbac-proxy-metric/0.log"
May 26 00:05:04.628836 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.628801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563d4d67-c7aa-43cc-b6f2-744f2607c29b/prom-label-proxy/0.log"
May 26 00:05:04.652589 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.652559    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563d4d67-c7aa-43cc-b6f2-744f2607c29b/init-config-reloader/0.log"
May 26 00:05:04.790192 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.790091    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6dfc7747c8-kxcqg_e0ceea22-b652-4c2b-ac89-f728cef64d58/metrics-server/0.log"
May 26 00:05:04.990838 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:04.990783    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nq27j_4b3d5441-69ab-4f3d-ac9f-4b2076699d8c/node-exporter/0.log"
May 26 00:05:05.009488 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.008981    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" path="/var/lib/kubelet/pods/8b60e57b-4fb0-4ab9-b25e-513a353cd911/volumes"
May 26 00:05:05.013366 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.013342    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nq27j_4b3d5441-69ab-4f3d-ac9f-4b2076699d8c/kube-rbac-proxy/0.log"
May 26 00:05:05.038915 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.038886    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nq27j_4b3d5441-69ab-4f3d-ac9f-4b2076699d8c/init-textfile/0.log"
May 26 00:05:05.359166 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.359137    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-prbm4_4b5d7cf7-ba46-4e2c-a437-7883d5c7fea4/prometheus-operator-admission-webhook/0.log"
May 26 00:05:05.391849 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.391816    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6846f6469c-gd6v8_63f26bc5-73a4-4b71-833a-c009538f5068/telemeter-client/0.log"
May 26 00:05:05.412207 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.412173    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6846f6469c-gd6v8_63f26bc5-73a4-4b71-833a-c009538f5068/reload/0.log"
May 26 00:05:05.432239 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.432205    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6846f6469c-gd6v8_63f26bc5-73a4-4b71-833a-c009538f5068/kube-rbac-proxy/0.log"
May 26 00:05:05.460302 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.460268    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c58f7bb8-xvqmp_ca119016-8532-4763-a583-37314c63f894/thanos-query/0.log"
May 26 00:05:05.480104 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.480074    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c58f7bb8-xvqmp_ca119016-8532-4763-a583-37314c63f894/kube-rbac-proxy-web/0.log"
May 26 00:05:05.501462 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.501395    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c58f7bb8-xvqmp_ca119016-8532-4763-a583-37314c63f894/kube-rbac-proxy/0.log"
May 26 00:05:05.523006 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.522973    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c58f7bb8-xvqmp_ca119016-8532-4763-a583-37314c63f894/prom-label-proxy/0.log"
May 26 00:05:05.542772 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.542696    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c58f7bb8-xvqmp_ca119016-8532-4763-a583-37314c63f894/kube-rbac-proxy-rules/0.log"
May 26 00:05:05.564003 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:05.563970    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c58f7bb8-xvqmp_ca119016-8532-4763-a583-37314c63f894/kube-rbac-proxy-metrics/0.log"
May 26 00:05:08.163899 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.163843    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"]
May 26 00:05:08.165184 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.164938    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" containerName="gather"
May 26 00:05:08.165184 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.164965    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" containerName="gather"
May 26 00:05:08.165184 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.164976    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" containerName="copy"
May 26 00:05:08.165184 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.164984    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" containerName="copy"
May 26 00:05:08.165184 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.165059    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" containerName="gather"
May 26 00:05:08.165184 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.165072    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="8b60e57b-4fb0-4ab9-b25e-513a353cd911" containerName="copy"
May 26 00:05:08.168641 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.168612    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.175550 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.175519    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"]
May 26 00:05:08.281531 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.281498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-lib-modules\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.281531 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.281534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6nn8\" (UniqueName: \"kubernetes.io/projected/a6f59018-46c6-4827-bd8b-856ab0e40392-kube-api-access-x6nn8\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.281752 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.281632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-sys\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.281752 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.281669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-proc\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.281752 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.281684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-podres\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.384018 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.383031    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-sys\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.384018 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.383094    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-proc\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.384018 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.383122    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-podres\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.384018 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.383194    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-lib-modules\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.384018 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.383221    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x6nn8\" (UniqueName: \"kubernetes.io/projected/a6f59018-46c6-4827-bd8b-856ab0e40392-kube-api-access-x6nn8\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.384018 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.383675    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-sys\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.384018 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.383768    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-proc\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.384018 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.383888    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-podres\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.384018 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.383988    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6f59018-46c6-4827-bd8b-856ab0e40392-lib-modules\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.391167 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.391134    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6nn8\" (UniqueName: \"kubernetes.io/projected/a6f59018-46c6-4827-bd8b-856ab0e40392-kube-api-access-x6nn8\") pod \"perf-node-gather-daemonset-lst8z\" (UID: \"a6f59018-46c6-4827-bd8b-856ab0e40392\") " pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.481569 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.481481    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:08.549179 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.549136    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wkpwv_122dfc84-a6fe-4de1-9b59-7b4fe23c4d56/dns/0.log"
May 26 00:05:08.571171 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.571141    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wkpwv_122dfc84-a6fe-4de1-9b59-7b4fe23c4d56/kube-rbac-proxy/0.log"
May 26 00:05:08.593441 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.593409    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fnfhv_a5c0947b-069f-4202-bfff-2b66c658a943/dns-node-resolver/0.log"
May 26 00:05:08.615753 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:08.615700    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"]
May 26 00:05:08.619026 ip-10-0-130-28 kubenswrapper[2578]: W0526 00:05:08.618995    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda6f59018_46c6_4827_bd8b_856ab0e40392.slice/crio-d86da6a86c22c02e23fe8ec20a1305aef5b514adad4a94a97bdb8cd0fdca91ca WatchSource:0}: Error finding container d86da6a86c22c02e23fe8ec20a1305aef5b514adad4a94a97bdb8cd0fdca91ca: Status 404 returned error can't find the container with id d86da6a86c22c02e23fe8ec20a1305aef5b514adad4a94a97bdb8cd0fdca91ca
May 26 00:05:09.006118 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:09.006089    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29662560-v8wdt_7dbada13-a66e-4b9a-8f4c-8be6289650b7/image-pruner/0.log"
May 26 00:05:09.046581 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:09.046544    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-695546cc48-lqqmc_5079703c-b039-467d-9bde-f1b9f5768bc5/registry/0.log"
May 26 00:05:09.088023 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:09.087993    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qhm9s_5783a810-60b3-470b-a71e-98357f95b962/node-ca/0.log"
May 26 00:05:09.569664 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:09.569619    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z" event={"ID":"a6f59018-46c6-4827-bd8b-856ab0e40392","Type":"ContainerStarted","Data":"fb47e5e5e14da1d61c53ab618869943fa6c5f99c99f51ddddfb911b8cf83d7ca"}
May 26 00:05:09.569664 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:09.569669    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z" event={"ID":"a6f59018-46c6-4827-bd8b-856ab0e40392","Type":"ContainerStarted","Data":"d86da6a86c22c02e23fe8ec20a1305aef5b514adad4a94a97bdb8cd0fdca91ca"}
May 26 00:05:09.570583 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:09.570556    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:09.587618 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:09.587563    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z" podStartSLOduration=1.5875462329999999 podStartE2EDuration="1.587546233s" podCreationTimestamp="2026-05-26 00:05:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 00:05:09.586282765 +0000 UTC m=+1809.141227026" watchObservedRunningTime="2026-05-26 00:05:09.587546233 +0000 UTC m=+1809.142490494"
May 26 00:05:10.176124 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:10.176089    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t89ch_facb9575-57af-4a8d-8e9e-453d4ad52215/serve-healthcheck-canary/0.log"
May 26 00:05:10.561841 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:10.561809    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4gjmv_eef71f49-b051-439d-91ef-3eb513154b1f/kube-rbac-proxy/0.log"
May 26 00:05:10.583389 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:10.583330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4gjmv_eef71f49-b051-439d-91ef-3eb513154b1f/exporter/0.log"
May 26 00:05:10.603711 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:10.603669    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4gjmv_eef71f49-b051-439d-91ef-3eb513154b1f/extractor/0.log"
May 26 00:05:15.732425 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:15.732393    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2c7q_25656d22-e01a-49b8-b274-0514816aae9c/kube-multus-additional-cni-plugins/0.log"
May 26 00:05:15.751890 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:15.751858    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2c7q_25656d22-e01a-49b8-b274-0514816aae9c/egress-router-binary-copy/0.log"
May 26 00:05:15.772077 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:15.772043    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2c7q_25656d22-e01a-49b8-b274-0514816aae9c/cni-plugins/0.log"
May 26 00:05:15.790633 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:15.790603    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2c7q_25656d22-e01a-49b8-b274-0514816aae9c/bond-cni-plugin/0.log"
May 26 00:05:15.808583 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:15.808551    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2c7q_25656d22-e01a-49b8-b274-0514816aae9c/routeoverride-cni/0.log"
May 26 00:05:15.828049 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:15.827973    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2c7q_25656d22-e01a-49b8-b274-0514816aae9c/whereabouts-cni-bincopy/0.log"
May 26 00:05:15.846198 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:15.846169    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2c7q_25656d22-e01a-49b8-b274-0514816aae9c/whereabouts-cni/0.log"
May 26 00:05:16.071202 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.071168    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-w7pqc_ebb49344-a3fe-4a42-8075-62ead776d565/kube-multus/0.log"
May 26 00:05:16.133658 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.133630    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lwvj8_accf65f4-6bda-4e39-ba44-c92d75f21e64/network-metrics-daemon/0.log"
May 26 00:05:16.154519 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.154486    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lwvj8_accf65f4-6bda-4e39-ba44-c92d75f21e64/kube-rbac-proxy/0.log"
May 26 00:05:16.587076 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.587047    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8rpn/perf-node-gather-daemonset-lst8z"
May 26 00:05:16.841783 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.841692    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-controller/0.log"
May 26 00:05:16.857557 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.857525    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/0.log"
May 26 00:05:16.874262 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.874215    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovn-acl-logging/1.log"
May 26 00:05:16.893508 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.893477    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/kube-rbac-proxy-node/0.log"
May 26 00:05:16.913289 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.913245    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/kube-rbac-proxy-ovn-metrics/0.log"
May 26 00:05:16.930757 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.930713    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/northd/0.log"
May 26 00:05:16.948576 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.948548    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/nbdb/0.log"
May 26 00:05:16.969776 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:16.969749    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/sbdb/0.log"
May 26 00:05:17.191221 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:17.191136    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9d2_0ab4c691-103b-4095-a962-d3c780650959/ovnkube-controller/0.log"
May 26 00:05:18.919696 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:18.919665    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zmrpl_d6f3d6c8-2003-4ce2-9c0d-4ae0bb00a94d/network-check-target-container/0.log"
May 26 00:05:20.032349 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:20.032317    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-lklf2_a27e83fe-fcf9-468d-ba60-ebc0746bc868/iptables-alerter/0.log"
May 26 00:05:20.634206 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:20.634171    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vr5m6_0fd250b8-53ba-4220-847a-0eba50daa2d7/tuned/0.log"
May 26 00:05:23.937579 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:23.937550    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xt6ft_d2a5c5ab-1582-42e5-9665-3a195365bb18/csi-driver/0.log"
May 26 00:05:23.960162 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:23.960130    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xt6ft_d2a5c5ab-1582-42e5-9665-3a195365bb18/csi-node-driver-registrar/0.log"
May 26 00:05:23.978300 ip-10-0-130-28 kubenswrapper[2578]: I0526 00:05:23.978268    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xt6ft_d2a5c5ab-1582-42e5-9665-3a195365bb18/csi-liveness-probe/0.log"