Jul 28 05:22:15.448547 ip-10-0-137-238 systemd[1]: Starting Kubernetes Kubelet... Jul 28 05:22:15.802520 ip-10-0-137-238 kubenswrapper[2529]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 05:22:15.802520 ip-10-0-137-238 kubenswrapper[2529]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 05:22:15.802520 ip-10-0-137-238 kubenswrapper[2529]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 05:22:15.802520 ip-10-0-137-238 kubenswrapper[2529]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 05:22:15.802520 ip-10-0-137-238 kubenswrapper[2529]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 05:22:15.806664 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.806567 2529 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 05:22:15.812099 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812073 2529 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:15.812099 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812095 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:15.812099 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812099 2529 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:15.812099 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812103 2529 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:15.812099 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812106 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812109 2529 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812114 2529 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812118 2529 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812121 2529 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812124 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812127 2529 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812130 2529 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812133 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812136 2529 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812139 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812141 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812144 2529 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812146 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812149 2529 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812151 2529 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812154 2529 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812157 2529 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812159 2529 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812162 2529 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:15.812337 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812165 2529 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812167 2529 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812170 2529 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812173 2529 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812175 2529 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812178 2529 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812181 2529 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812184 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812187 2529 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812191 2529 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812195 2529 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812199 2529 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812202 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812205 2529 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812220 2529 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812223 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812226 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812229 2529 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812231 2529 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812234 2529 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:15.812812 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812238 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812241 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812244 2529 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812246 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812249 2529 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812252 2529 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812254 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812258 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812263 2529 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812268 2529 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812271 2529 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812274 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812277 2529 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812280 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812283 2529 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812286 2529 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812289 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812292 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812294 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:15.813332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812297 2529 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812300 2529 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812303 2529 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812305 2529 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812308 2529 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812310 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812313 2529 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812316 2529 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812319 2529 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812322 2529 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812324 2529 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812327 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812329 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812332 2529 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812335 2529 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812337 2529 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812340 2529 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812343 2529 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812346 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812350 2529 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:15.813789 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812355 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812358 2529 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812360 2529 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812363 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812366 2529 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812369 2529 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812372 2529 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812374 2529 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812377 2529 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812380 2529 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812382 2529 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812385 2529 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812387 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812390 2529 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812393 2529 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.812396 2529 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813132 2529 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813139 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813143 2529 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813146 2529 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:15.814294 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813148 2529 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813151 2529 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813154 2529 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813157 2529 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813159 2529 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813162 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813165 2529 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813168 2529 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813171 2529 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813173 2529 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813176 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813180 2529 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813184 2529 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813189 2529 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813192 2529 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813195 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813199 2529 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813202 2529 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813205 2529 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:15.814762 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813223 2529 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813226 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813228 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813231 2529 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813233 2529 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813237 2529 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813239 2529 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813243 2529 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813245 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813248 2529 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813251 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813254 2529 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813257 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813260 2529 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813263 2529 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813265 2529 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813268 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813271 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813274 2529 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813276 2529 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:15.815264 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813279 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813282 2529 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813284 2529 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813287 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813289 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813292 2529 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813294 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813298 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813301 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813303 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813306 2529 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813308 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813311 2529 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813313 2529 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813316 2529 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813318 2529 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813321 2529 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813323 2529 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813326 2529 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813328 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:15.815750 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813331 2529 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813334 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813336 2529 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813340 2529 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813344 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813348 2529 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813350 2529 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813353 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813356 2529 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813358 2529 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813361 2529 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813363 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813366 2529 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813369 2529 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813371 2529 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813374 2529 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813376 2529 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813378 2529 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813381 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813384 2529 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:15.816256 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813390 2529 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813393 2529 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813395 2529 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813398 2529 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813401 2529 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813404 2529 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813408 2529 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813411 2529 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813414 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813417 2529 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813420 2529 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813423 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813425 2529 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813428 2529 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813430 2529 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813433 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813835 2529 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813841 2529 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813844 2529 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:15.816760 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813848 2529 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813851 2529 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813854 2529 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813858 2529 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813861 2529 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813864 2529 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813867 2529 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813869 2529 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813872 2529 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813874 2529 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813877 2529 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813880 2529 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813882 2529 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813885 2529 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813888 2529 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813891 2529 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813894 2529 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813896 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813899 2529 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:15.817268 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813902 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813904 2529 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813907 2529 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813909 2529 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813912 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813915 2529 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813917 2529 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813920 2529 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813923 2529 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813926 2529 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813928 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813931 2529 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813934 2529 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813937 2529 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813939 2529 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813942 2529 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813944 2529 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813947 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813949 2529 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813952 2529 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:15.817776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813954 2529 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813958 2529 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813961 2529 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813964 2529 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813967 2529 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813969 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813972 2529 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813975 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813977 2529 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813980 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813983 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813985 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813988 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813990 2529 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813993 2529 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813995 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.813998 2529 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814000 2529 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814003 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:15.818272 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814005 2529 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814008 2529 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814011 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814013 2529 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814016 2529 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814018 2529 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814021 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814023 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814026 2529 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814029 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814031 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814034 2529 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814036 2529 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814038 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814041 2529 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814043 2529 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814046 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814049 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814051 2529 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814053 2529 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:15.818742 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814056 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814059 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814061 2529 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814066 2529 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814069 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814072 2529 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814075 2529 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814078 2529 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814081 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814083 2529 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814086 2529 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814089 2529 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814091 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814094 2529 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814097 2529 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814100 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814103 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814105 2529 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814171 2529 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814185 2529 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814192 2529 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 05:22:15.819285 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814197 2529 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814201 2529 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814205 2529 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814222 2529 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814227 2529 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814231 2529 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814234 2529 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814237 2529 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814241 2529 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814244 2529 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814247 2529 flags.go:64] FLAG: --cgroup-root="" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814263 2529 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814267 2529 flags.go:64] FLAG: --client-ca-file="" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814271 2529 flags.go:64] FLAG: --cloud-provider="external" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814274 2529 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814280 2529 flags.go:64] FLAG: --cluster-domain="" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814283 2529 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814287 2529 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814290 2529 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814293 2529 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814298 2529 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814301 2529 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814304 2529 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 05:22:15.819878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814307 2529 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814311 2529 flags.go:64] FLAG: --contention-profiling="false" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814314 2529 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814317 2529 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814320 2529 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814323 2529 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814327 2529 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814330 2529 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814333 2529 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814336 2529 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814339 2529 flags.go:64] FLAG: --enable-server="true" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814342 2529 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814346 2529 flags.go:64] FLAG: --event-burst="100" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814350 2529 flags.go:64] FLAG: --event-qps="50" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814353 2529 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814356 2529 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814359 2529 flags.go:64] FLAG: --eviction-hard="" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814363 2529 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814365 2529 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814368 2529 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814371 2529 flags.go:64] FLAG: --eviction-soft="" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814374 2529 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814377 2529 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814381 2529 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814385 2529 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 05:22:15.820492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814388 2529 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814391 2529 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814393 2529 flags.go:64] FLAG: --feature-gates="" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814397 2529 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814402 2529 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814406 2529 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814409 2529 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814412 2529 flags.go:64] FLAG: --healthz-port="10248" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814415 2529 flags.go:64] FLAG: --help="false" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814418 2529 flags.go:64] FLAG: --hostname-override="ip-10-0-137-238.ec2.internal" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814421 2529 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814424 2529 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814427 2529 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814431 2529 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814434 2529 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814437 2529 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814440 2529 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814443 2529 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814446 2529 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814449 2529 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814452 2529 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814455 2529 flags.go:64] FLAG: --kube-reserved="" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814457 2529 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814460 2529 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 05:22:15.821273 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814464 2529 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814466 2529 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814469 2529 flags.go:64] FLAG: --lock-file="" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814472 2529 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814475 2529 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814478 2529 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814483 2529 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814487 2529 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814490 2529 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814492 2529 flags.go:64] FLAG: --logging-format="text" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814495 2529 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814499 2529 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814505 2529 flags.go:64] FLAG: --manifest-url="" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814509 2529 flags.go:64] FLAG: --manifest-url-header="" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814513 2529 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814516 2529 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814520 2529 flags.go:64] FLAG: --max-pods="110" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814523 2529 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814526 2529 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814529 2529 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814532 2529 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814535 2529 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814538 2529 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814541 2529 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814548 2529 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 05:22:15.822341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814551 2529 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814554 2529 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814557 2529 flags.go:64] FLAG: --pod-cidr="" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814560 2529 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814566 2529 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814568 2529 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814571 2529 flags.go:64] FLAG: --pods-per-core="0" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814574 2529 flags.go:64] FLAG: --port="10250" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814578 2529 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814581 2529 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ec189e61f211559a" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814584 2529 flags.go:64] FLAG: --qos-reserved="" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814587 2529 flags.go:64] FLAG: --read-only-port="10255" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814590 2529 flags.go:64] FLAG: --register-node="true" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814595 2529 flags.go:64] FLAG: --register-with-taints="" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814599 2529 flags.go:64] FLAG: --registry-burst="10" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814602 2529 flags.go:64] FLAG: --registry-qps="5" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814605 2529 flags.go:64] FLAG: --reserved-cpus="" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814608 2529 flags.go:64] FLAG: --reserved-memory="" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814612 2529 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814615 2529 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814620 2529 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814623 2529 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814625 2529 flags.go:64] FLAG: --runonce="false" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814628 2529 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814631 2529 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 05:22:15.823385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814634 2529 flags.go:64] FLAG: --seccomp-default="false" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814637 2529 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814640 2529 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814642 2529 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814645 2529 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814649 2529 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814652 2529 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814654 2529 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814657 2529 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814660 2529 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814664 2529 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814666 2529 flags.go:64] FLAG: --system-cgroups="" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814669 2529 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814675 2529 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814677 2529 flags.go:64] FLAG: --tls-cert-file="" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814680 2529 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814685 2529 flags.go:64] FLAG: --tls-min-version="" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814687 2529 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814690 2529 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814693 2529 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814697 2529 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814700 2529 flags.go:64] FLAG: --v="2" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814704 2529 flags.go:64] FLAG: --version="false" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814709 2529 flags.go:64] FLAG: --vmodule="" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814712 2529 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 05:22:15.824031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.814716 2529 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814833 2529 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814839 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814842 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814845 2529 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814849 2529 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814853 2529 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814862 2529 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814866 2529 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814870 2529 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814872 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814875 2529 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814878 2529 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814880 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814883 2529 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814886 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814889 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814893 2529 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814896 2529 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:15.825126 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814899 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814902 2529 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814904 2529 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814907 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814909 2529 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814912 2529 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814915 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814918 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814922 2529 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814925 2529 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814928 2529 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814930 2529 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814934 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814937 2529 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814940 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814944 2529 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814947 2529 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814950 2529 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814952 2529 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:15.825845 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814955 2529 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814957 2529 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814960 2529 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814963 2529 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814965 2529 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814968 2529 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814970 2529 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814973 2529 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814975 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814979 2529 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814982 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814985 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814988 2529 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814991 2529 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814994 2529 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814996 2529 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.814999 2529 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815001 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815004 2529 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:15.826629 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815006 2529 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815009 2529 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815012 2529 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815015 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815018 2529 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815020 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815023 2529 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815026 2529 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815029 2529 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815033 2529 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815036 2529 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815039 2529 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815041 2529 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815044 2529 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815046 2529 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815049 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815051 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815054 2529 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815057 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815059 2529 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:15.827440 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815062 2529 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815064 2529 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815067 2529 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815069 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815072 2529 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815075 2529 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815077 2529 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815080 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815082 2529 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815085 2529 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815087 2529 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815090 2529 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815093 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815095 2529 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815099 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815102 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815105 2529 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815107 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815110 2529 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815113 2529 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:15.828040 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815116 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815120 2529 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.815122 2529 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.815661 2529 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.823265 2529 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.823292 2529 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.824030 2529 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.824045 2529 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824142 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824152 2529 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824157 2529 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824162 2529 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824167 2529 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824172 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824177 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:15.828690 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824181 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824186 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824191 2529 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824195 2529 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824200 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824205 2529 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824231 2529 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824236 2529 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824240 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824244 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824248 2529 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824252 2529 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824256 2529 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824260 2529 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824264 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824269 2529 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824273 2529 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824277 2529 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824281 2529 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824285 2529 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:15.829083 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824290 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824294 2529 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824300 2529 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824304 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824309 2529 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824313 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824317 2529 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824321 2529 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824325 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824330 2529 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824333 2529 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824338 2529 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824342 2529 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824347 2529 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824352 2529 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824356 2529 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824360 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824364 2529 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824368 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824372 2529 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:15.829673 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824377 2529 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824381 2529 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824386 2529 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824390 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824394 2529 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824398 2529 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824402 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824406 2529 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824410 2529 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824414 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824419 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824423 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824430 2529 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824436 2529 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824442 2529 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824447 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824452 2529 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824456 2529 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824461 2529 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:15.830263 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824465 2529 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824469 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824473 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824477 2529 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824482 2529 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824488 2529 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824493 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824498 2529 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824501 2529 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824506 2529 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824510 2529 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824514 2529 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824518 2529 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824522 2529 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824526 2529 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824530 2529 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824535 2529 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824539 2529 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824544 2529 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:15.830782 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824548 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824552 2529 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824557 2529 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824561 2529 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824565 2529 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824569 2529 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824574 2529 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824578 2529 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824582 2529 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824587 2529 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824594 2529 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824600 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824605 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824610 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:15.831521 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.824619 2529 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824814 2529 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824821 2529 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824826 2529 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824831 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824836 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824840 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824844 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824849 2529 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824853 2529 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824858 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824862 2529 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824867 2529 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824871 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824876 2529 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824880 2529 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824884 2529 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824889 2529 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824893 2529 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824898 2529 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:15.832081 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824902 2529 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824906 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824911 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824915 2529 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824919 2529 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824924 2529 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824929 2529 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824933 2529 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824937 2529 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824950 2529 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824955 2529 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824959 2529 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824963 2529 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824968 2529 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824972 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824976 2529 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824981 2529 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824986 2529 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824991 2529 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:15.832927 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.824995 2529 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825002 2529 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825007 2529 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825012 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825016 2529 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825021 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825025 2529 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825029 2529 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825034 2529 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825038 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825042 2529 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825046 2529 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825050 2529 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825055 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825059 2529 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825063 2529 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825068 2529 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825072 2529 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825076 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825080 2529 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:15.833719 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825085 2529 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825089 2529 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825094 2529 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825100 2529 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825104 2529 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825108 2529 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825114 2529 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825119 2529 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825124 2529 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825128 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825133 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825136 2529 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825140 2529 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825145 2529 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825149 2529 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825153 2529 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825157 2529 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825161 2529 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825165 2529 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:15.834273 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825169 2529 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825173 2529 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825177 2529 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825182 2529 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825186 2529 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825190 2529 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825194 2529 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825199 2529 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825203 2529 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825222 2529 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825227 2529 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825231 2529 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825236 2529 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825241 2529 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825245 2529 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825250 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825254 2529 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825259 2529 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825264 2529 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825268 2529 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:15.834779 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825272 2529 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:15.835332 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:15.825277 2529 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:15.835332 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.825286 2529 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 05:22:15.835332 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.825478 2529 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 05:22:15.835332 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.828105 2529 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 05:22:15.835332 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.829372 2529 server.go:1019] "Starting client certificate rotation" Jul 28 05:22:15.835332 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.829474 2529 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 05:22:15.835332 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.830152 2529 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 05:22:15.835332 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.831901 2529 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 05:22:15.835332 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.833728 2529 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 05:22:15.844303 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.844283 2529 log.go:25] "Validated CRI v1 runtime API" Jul 28 05:22:15.846205 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.846187 2529 log.go:25] "Validated CRI v1 image API" Jul 28 05:22:15.847073 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.847044 2529 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 05:22:15.849641 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.849620 2529 fs.go:135] Filesystem UUIDs: map[4730fbed-327d-45fc-b8e6-2a121725d666:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8df638b4-0279-4522-946d-1acb78ccba86:/dev/nvme0n1p4] Jul 28 05:22:15.849727 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.849639 2529 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 05:22:15.854743 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.854631 2529 manager.go:216] Machine: {Timestamp:2026-07-28 05:22:15.853356748 +0000 UTC m=+0.298697275 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109842 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c99dc0fd85ebb49aedf982fd40252 SystemUUID:ec2c99dc-0fd8-5ebb-49ae-df982fd40252 BootID:c682af1d-0521-4369-8a0c-8d3b5122e436 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:41:62:4a:fd:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:41:62:4a:fd:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:5a:23:58:eb:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 05:22:15.854743 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.854741 2529 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 05:22:15.854879 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.854826 2529 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 05:22:15.858589 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.858561 2529 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 05:22:15.858746 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.858594 2529 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-238.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 05:22:15.859374 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.859358 2529 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 05:22:15.859725 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.859713 2529 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 05:22:15.859725 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.859729 2529 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 05:22:15.859795 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.859736 2529 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 05:22:15.860445 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.860435 2529 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 05:22:15.861238 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.861227 2529 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 05:22:15.862954 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.862944 2529 state_mem.go:36] "Initialized new in-memory state store" Jul 28 05:22:15.863074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.863065 2529 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 05:22:15.863105 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.863097 2529 kubelet.go:486] "Attempting to sync node with API server" Jul 28 05:22:15.863132 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.863106 2529 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 05:22:15.863132 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.863120 2529 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 05:22:15.863132 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.863129 2529 kubelet.go:398] "Adding apiserver pod source" Jul 28 05:22:15.863247 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.863138 2529 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 05:22:15.863559 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.863539 2529 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 05:22:15.863604 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.863567 2529 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 05:22:15.865608 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.865586 2529 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 05:22:15.867691 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.867677 2529 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 05:22:15.867743 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.867708 2529 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 05:22:15.868972 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.868958 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 05:22:15.869006 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.868978 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 05:22:15.869006 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.868984 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 05:22:15.869006 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.868989 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 05:22:15.869006 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.868997 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 05:22:15.869006 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.869002 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 05:22:15.869133 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.869010 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 05:22:15.869133 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.869015 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 05:22:15.869133 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.869022 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 05:22:15.869133 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.869027 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 05:22:15.869133 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.869035 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 05:22:15.869133 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.869040 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 05:22:15.870250 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.870239 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 05:22:15.870250 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.870250 2529 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 05:22:15.872623 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.872597 2529 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-238.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 05:22:15.873247 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:15.873229 2529 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 05:22:15.873247 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:15.873232 2529 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 05:22:15.873585 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.873573 2529 server.go:1268] "Started kubelet" Jul 28 05:22:15.873662 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.873626 2529 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 05:22:15.873812 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.873738 2529 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 05:22:15.877639 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.877593 2529 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 05:22:15.878669 ip-10-0-137-238 systemd[1]: Started Kubernetes Kubelet. Jul 28 05:22:15.879933 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.879904 2529 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d7t6m" Jul 28 05:22:15.880222 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.880189 2529 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 05:22:15.881355 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:15.881327 2529 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 05:22:15.882574 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.882557 2529 server.go:310] "Adding debug handlers to kubelet server" Jul 28 05:22:15.884901 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.884884 2529 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d7t6m" Jul 28 05:22:15.885960 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:15.885064 2529 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-238.ec2.internal.18c65c9dfa09b390 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-238.ec2.internal,UID:ip-10-0-137-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-238.ec2.internal,},FirstTimestamp:2026-07-28 05:22:15.873541008 +0000 UTC m=+0.318881539,LastTimestamp:2026-07-28 05:22:15.873541008 +0000 UTC m=+0.318881539,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-238.ec2.internal,}" Jul 28 05:22:15.888492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.888472 2529 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 05:22:15.888492 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.888478 2529 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 05:22:15.888625 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:15.888550 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:15.888941 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.888922 2529 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 05:22:15.888941 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.888948 2529 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 05:22:15.889150 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.888982 2529 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 05:22:15.889150 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.889064 2529 reconstruct.go:97] "Volume reconstruction finished" Jul 28 05:22:15.889301 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.889160 2529 reconciler.go:29] "Reconciler: start to sync state" Jul 28 05:22:15.889377 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.889362 2529 factory.go:55] Registering systemd factory Jul 28 05:22:15.889456 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.889382 2529 factory.go:223] Registration of the systemd container factory successfully Jul 28 05:22:15.889677 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.889655 2529 factory.go:153] Registering CRI-O factory Jul 28 05:22:15.889677 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.889673 2529 factory.go:223] Registration of the crio container factory successfully Jul 28 05:22:15.889780 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.889715 2529 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 05:22:15.889780 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.889733 2529 factory.go:103] Registering Raw factory Jul 28 05:22:15.889780 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.889744 2529 manager.go:1196] Started watching for new ooms in manager Jul 28 05:22:15.890586 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.890570 2529 manager.go:349] Starting recovery of all containers Jul 28 05:22:15.897451 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.897428 2529 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:15.898160 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.898117 2529 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 05:22:15.902558 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:15.900509 2529 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-238.ec2.internal\" not found" node="ip-10-0-137-238.ec2.internal" Jul 28 05:22:15.902558 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.901718 2529 manager.go:354] Recovery completed Jul 28 05:22:15.906551 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.906539 2529 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:15.908430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.908412 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:15.908523 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.908442 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:15.908523 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.908454 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:15.909033 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.909018 2529 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 05:22:15.909033 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.909030 2529 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 05:22:15.909125 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.909048 2529 state_mem.go:36] "Initialized new in-memory state store" Jul 28 05:22:15.909125 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.909057 2529 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:15.909799 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.909782 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:15.909870 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.909808 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:15.909870 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.909818 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:15.910546 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.910532 2529 policy_none.go:49] "None policy: Start" Jul 28 05:22:15.910601 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.910550 2529 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 05:22:15.910601 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.910560 2529 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 05:22:15.911540 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.911523 2529 policy_none.go:47] "Start" Jul 28 05:22:15.945240 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.945027 2529 manager.go:337] "Starting Device Plugin manager" Jul 28 05:22:15.945393 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:15.945278 2529 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 05:22:15.945393 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.945290 2529 server.go:87] "Starting device plugin registration server" Jul 28 05:22:15.945527 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.945515 2529 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 05:22:15.945581 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.945527 2529 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 05:22:15.945683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.945669 2529 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 05:22:15.945774 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.945762 2529 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 05:22:15.945827 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.945775 2529 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 05:22:15.946363 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:15.946346 2529 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 05:22:15.946475 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:15.946396 2529 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:15.960890 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:15.960868 2529 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2567 into the system container \"/system.slice\": " Jul 28 05:22:16.045653 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.045624 2529 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:16.046849 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.046694 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:16.046849 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.046725 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:16.046849 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.046739 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:16.046849 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.046766 2529 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.054277 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.054259 2529 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.054356 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.054284 2529 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-238.ec2.internal\": node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.066285 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.066254 2529 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 05:22:16.066405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.066296 2529 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 05:22:16.066405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.066317 2529 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 05:22:16.066405 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.066350 2529 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 05:22:16.068442 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.068423 2529 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:16.069520 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.069504 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.166942 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.166886 2529 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal"] Jul 28 05:22:16.167131 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.166992 2529 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:16.168604 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.168587 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:16.168676 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.168617 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:16.168676 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.168628 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:16.169665 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.169650 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.169709 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.169668 2529 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:16.169840 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.169827 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.169878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.169855 2529 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:16.170408 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.170393 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:16.170487 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.170420 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:16.170487 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.170430 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:16.170487 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.170400 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:16.170577 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.170496 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:16.170577 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.170508 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:16.171516 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.171504 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.171570 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.171529 2529 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:16.172239 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.172206 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:16.172320 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.172256 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:16.172320 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.172270 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:16.185753 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.185738 2529 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-238.ec2.internal\" not found" node="ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.189947 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.189925 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.190014 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.189953 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.190014 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.189954 2529 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-238.ec2.internal\" not found" node="ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.190014 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.189990 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d68380c15b9e00b00440ef022ddd164-config\") pod \"kube-apiserver-proxy-ip-10-0-137-238.ec2.internal\" (UID: \"6d68380c15b9e00b00440ef022ddd164\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.270295 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.270272 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.290081 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.290052 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.290195 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.290109 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.290195 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.290119 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.290195 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.290136 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d68380c15b9e00b00440ef022ddd164-config\") pod \"kube-apiserver-proxy-ip-10-0-137-238.ec2.internal\" (UID: \"6d68380c15b9e00b00440ef022ddd164\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.290195 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.290188 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.290340 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.290196 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d68380c15b9e00b00440ef022ddd164-config\") pod \"kube-apiserver-proxy-ip-10-0-137-238.ec2.internal\" (UID: \"6d68380c15b9e00b00440ef022ddd164\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.370635 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.370551 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.471041 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.471009 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.488154 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.488134 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.494053 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.494035 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" Jul 28 05:22:16.571917 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.571878 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.672435 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.672364 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.743679 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.743657 2529 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:16.773312 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.773283 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.829800 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.829777 2529 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 05:22:16.830205 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.829916 2529 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 05:22:16.830205 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.829927 2529 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 05:22:16.830205 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.829948 2529 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 05:22:16.873382 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:16.873354 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:16.889339 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.889294 2529 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 05:17:17 +0000 UTC" deadline="2028-02-21 13:29:56.198400792 +0000 UTC" Jul 28 05:22:16.889339 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:16.889337 2529 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13760h7m39.309067814s" Jul 28 05:22:18.487161 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:18.487137 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:18.526530 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:18.526487 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a91ffb87b056369246a9445ebaa0cf.slice/crio-8e3ca41f493673e2c7ae7696ea9ec3bb6d606a0082b43ff2411700da635d1b7d WatchSource:0}: Error finding container 8e3ca41f493673e2c7ae7696ea9ec3bb6d606a0082b43ff2411700da635d1b7d: Status 404 returned error can't find the container with id 8e3ca41f493673e2c7ae7696ea9ec3bb6d606a0082b43ff2411700da635d1b7d Jul 28 05:22:18.530687 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:18.530668 2529 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 05:22:18.549353 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:18.549324 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d68380c15b9e00b00440ef022ddd164.slice/crio-9d8d286a067f41a2d06aa2931ff5997309acf1d9ae5ace03b4232e58171b7c98 WatchSource:0}: Error finding container 9d8d286a067f41a2d06aa2931ff5997309acf1d9ae5ace03b4232e58171b7c98: Status 404 returned error can't find the container with id 9d8d286a067f41a2d06aa2931ff5997309acf1d9ae5ace03b4232e58171b7c98 Jul 28 05:22:18.582323 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:18.582279 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" event={"ID":"67a91ffb87b056369246a9445ebaa0cf","Type":"ContainerStarted","Data":"8e3ca41f493673e2c7ae7696ea9ec3bb6d606a0082b43ff2411700da635d1b7d"} Jul 28 05:22:18.583331 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:18.583306 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" event={"ID":"6d68380c15b9e00b00440ef022ddd164","Type":"ContainerStarted","Data":"9d8d286a067f41a2d06aa2931ff5997309acf1d9ae5ace03b4232e58171b7c98"} Jul 28 05:22:18.587456 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:18.587440 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:18.687930 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:18.687903 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:18.788461 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:18.788397 2529 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found" Jul 28 05:22:18.816980 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:18.816960 2529 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:18.902021 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:18.901990 2529 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" Jul 28 05:22:18.911242 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:18.911217 2529 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 05:22:18.912118 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:18.912098 2529 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" Jul 28 05:22:18.919245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:18.919222 2529 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 05:22:19.186715 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.186636 2529 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:19.377796 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.377762 2529 apiserver.go:52] "Watching apiserver" Jul 28 05:22:19.387035 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.387011 2529 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 05:22:19.387939 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.387914 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-qhsdk","openshift-cluster-node-tuning-operator/tuned-4wzrz","openshift-dns/node-resolver-6cnbv","openshift-image-registry/node-ca-z6ff9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal","openshift-network-diagnostics/network-check-target-srm2m","kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l","openshift-multus/multus-additional-cni-plugins-xxvrw","openshift-multus/multus-ctmks","openshift-multus/network-metrics-daemon-qntsm","openshift-network-operator/iptables-alerter-7rc59","openshift-ovn-kubernetes/ovnkube-node-gpmnb","kube-system/global-pull-secret-syncer-r52w4"] Jul 28 05:22:19.390593 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.390541 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.392760 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.392740 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 05:22:19.392886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.392784 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 05:22:19.392886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.392791 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 05:22:19.393022 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.393005 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:19.393022 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.393017 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5csr9\"" Jul 28 05:22:19.393095 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.393075 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:19.393202 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.393189 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 05:22:19.395209 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.395188 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.397123 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.397103 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 05:22:19.397393 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.397360 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.397468 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.397431 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ww2xb\"" Jul 28 05:22:19.397468 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.397445 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 05:22:19.399627 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.399326 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 05:22:19.399627 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.399608 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 05:22:19.399627 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.399628 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 05:22:19.399829 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.399679 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wv9s8\"" Jul 28 05:22:19.399829 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.399684 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:19.399829 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.399755 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:19.401730 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.401714 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:19.401826 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.401762 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:19.402671 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.402658 2529 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 05:22:19.404092 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.404074 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.406917 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.406195 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 05:22:19.406917 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.406564 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbkbk\"" Jul 28 05:22:19.406917 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.406597 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 05:22:19.409142 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.408353 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.410407 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.410388 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 05:22:19.410625 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.410607 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 05:22:19.410712 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.410677 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 05:22:19.410834 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.410817 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 05:22:19.410892 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.410860 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 05:22:19.410945 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.410933 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.411652 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.411635 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 05:22:19.411741 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.411709 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-85f2f\"" Jul 28 05:22:19.413172 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.413155 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vkqvs\"" Jul 28 05:22:19.413374 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.413359 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 05:22:19.413544 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.413476 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:19.413729 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.413550 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 05:22:19.413951 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.413921 2529 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 05:22:19.414264 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414246 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c86d6b2-647e-4409-9fb3-ca3861e7b9d4-serviceca\") pod \"node-ca-z6ff9\" (UID: \"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4\") " pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.414341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414279 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-os-release\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.414341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414306 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-socket-dir-parent\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.414341 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414332 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-run-netns\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.414488 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414359 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-cnibin\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.414488 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414384 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7wp4\" (UniqueName: \"kubernetes.io/projected/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-kube-api-access-b7wp4\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:19.414488 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414406 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7w2f\" (UniqueName: \"kubernetes.io/projected/9c86d6b2-647e-4409-9fb3-ca3861e7b9d4-kube-api-access-c7w2f\") pod \"node-ca-z6ff9\" (UID: \"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4\") " pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.414488 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414429 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.414488 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414467 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.414752 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414507 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlxl\" (UniqueName: \"kubernetes.io/projected/6e82bb4e-d528-49e5-8c0d-02eb57c2e399-kube-api-access-wdlxl\") pod \"node-resolver-6cnbv\" (UID: \"6e82bb4e-d528-49e5-8c0d-02eb57c2e399\") " pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.414752 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414655 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.414752 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414680 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-cni-dir\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.414752 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414703 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ce96221-80e8-4768-a905-818462349deb-dbus\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:19.414752 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414738 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:19.414982 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414769 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e82bb4e-d528-49e5-8c0d-02eb57c2e399-tmp-dir\") pod \"node-resolver-6cnbv\" (UID: \"6e82bb4e-d528-49e5-8c0d-02eb57c2e399\") " pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.414982 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414808 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-var-lib-kubelet\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.414982 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414836 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-conf-dir\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.414982 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414860 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-etc-kubernetes\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.414982 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414882 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.414982 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414927 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-var-lib-cni-multus\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.414982 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414945 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-hostroot\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.415314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.414989 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-run-multus-certs\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.415314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415024 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ce96221-80e8-4768-a905-818462349deb-kubelet-config\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:19.415314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415065 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-system-cni-dir\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.415314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415102 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6h5t\" (UniqueName: \"kubernetes.io/projected/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-kube-api-access-g6h5t\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.415314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415143 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-system-cni-dir\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.415314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415168 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-cnibin\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.415314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415206 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:19.415314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415239 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c86d6b2-647e-4409-9fb3-ca3861e7b9d4-host\") pod \"node-ca-z6ff9\" (UID: \"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4\") " pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.415314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415300 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:19.415712 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415320 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72c5583a-0e4e-42b6-b0d0-da18faa742ff-cni-binary-copy\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.415712 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415341 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-run-k8s-cni-cncf-io\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.415712 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415604 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-var-lib-cni-bin\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.415712 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415643 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtjg\" (UniqueName: \"kubernetes.io/projected/72c5583a-0e4e-42b6-b0d0-da18faa742ff-kube-api-access-4gtjg\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.415712 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415678 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e82bb4e-d528-49e5-8c0d-02eb57c2e399-hosts-file\") pod \"node-resolver-6cnbv\" (UID: \"6e82bb4e-d528-49e5-8c0d-02eb57c2e399\") " pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.415931 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415722 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-os-release\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.415931 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.415756 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-daemon-config\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.416079 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.416022 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 05:22:19.416896 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.416674 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hcwm9\"" Jul 28 05:22:19.417065 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.417041 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 05:22:19.419309 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.417533 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.420314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.420153 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 05:22:19.420407 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.420391 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tqh87\"" Jul 28 05:22:19.420470 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.420392 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 05:22:19.420552 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.420531 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 05:22:19.420619 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.420578 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.422753 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.422724 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 05:22:19.422819 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.422808 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 05:22:19.422906 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.422888 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6wvfb\"" Jul 28 05:22:19.422978 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.422943 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 05:22:19.434732 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.434711 2529 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xzmlc" Jul 28 05:22:19.442209 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.442190 2529 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xzmlc" Jul 28 05:22:19.503275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.503246 2529 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 05:22:19.516640 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516614 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0cf60e9-379a-4455-b9c5-6c72caa1774e-env-overrides\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.516798 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516672 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-socket-dir-parent\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.516798 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516775 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-run-netns\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.516908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516813 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-socket-dir-parent\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.516908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516816 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-cnibin\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.516908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516848 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.516908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516872 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-cnibin\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.516908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516873 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-run-netns\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.516908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516880 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-kubernetes\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516903 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-systemd\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516933 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/681f5ec2-4771-4e6a-b539-8b88971c205e-tmp\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516959 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-cni-bin\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.516984 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517011 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ce96221-80e8-4768-a905-818462349deb-dbus\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517035 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517059 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517083 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-var-lib-kubelet\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517105 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-conf-dir\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517128 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-etc-kubernetes\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517152 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-run\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517192 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-cni-netd\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.517245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517237 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-host\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517259 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-log-socket\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517286 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-system-cni-dir\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517312 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-system-cni-dir\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517315 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517340 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-cnibin\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517366 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517391 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-socket-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517419 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-tuned\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517454 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ce96221-80e8-4768-a905-818462349deb-dbus\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517455 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517442 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517499 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-var-lib-kubelet\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517510 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-run-k8s-cni-cncf-io\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517573 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-var-lib-cni-bin\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517574 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-system-cni-dir\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517587 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-system-cni-dir\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.517907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517599 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-var-lib-cni-bin\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517603 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3559246-3cfe-44e7-95ea-6a974178f96e-konnectivity-ca\") pod \"konnectivity-agent-qhsdk\" (UID: \"c3559246-3cfe-44e7-95ea-6a974178f96e\") " pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517621 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-etc-kubernetes\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517633 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-registration-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517653 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-run-k8s-cni-cncf-io\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.517658 2529 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517669 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-conf-dir\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517687 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-sysctl-conf\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.517701 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret podName:7ce96221-80e8-4768-a905-818462349deb nodeName:}" failed. No retries permitted until 2026-07-28 05:22:20.017686229 +0000 UTC m=+2.949721367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret") pod "global-pull-secret-syncer-r52w4" (UID: "7ce96221-80e8-4768-a905-818462349deb") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517721 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-cnibin\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517784 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0cf60e9-379a-4455-b9c5-6c72caa1774e-ovnkube-script-lib\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517805 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e82bb4e-d528-49e5-8c0d-02eb57c2e399-hosts-file\") pod \"node-resolver-6cnbv\" (UID: \"6e82bb4e-d528-49e5-8c0d-02eb57c2e399\") " pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517820 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-daemon-config\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517840 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-sys-fs\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517864 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-run-netns\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517876 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e82bb4e-d528-49e5-8c0d-02eb57c2e399-hosts-file\") pod \"node-resolver-6cnbv\" (UID: \"6e82bb4e-d528-49e5-8c0d-02eb57c2e399\") " pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517887 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-run-openvswitch\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.518582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517912 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0cf60e9-379a-4455-b9c5-6c72caa1774e-ovn-node-metrics-cert\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517936 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/262fafeb-4ba5-4b6e-a870-92f41b198b15-iptables-alerter-script\") pod \"iptables-alerter-7rc59\" (UID: \"262fafeb-4ba5-4b6e-a870-92f41b198b15\") " pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517957 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-sysctl-d\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.517984 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-lib-modules\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518002 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-etc-openvswitch\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518019 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7wp4\" (UniqueName: \"kubernetes.io/projected/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-kube-api-access-b7wp4\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518037 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7w2f\" (UniqueName: \"kubernetes.io/projected/9c86d6b2-647e-4409-9fb3-ca3861e7b9d4-kube-api-access-c7w2f\") pod \"node-ca-z6ff9\" (UID: \"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4\") " pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518059 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518095 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-var-lib-openvswitch\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518118 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518146 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlxl\" (UniqueName: \"kubernetes.io/projected/6e82bb4e-d528-49e5-8c0d-02eb57c2e399-kube-api-access-wdlxl\") pod \"node-resolver-6cnbv\" (UID: \"6e82bb4e-d528-49e5-8c0d-02eb57c2e399\") " pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518168 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-cni-dir\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518192 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd96s\" (UniqueName: \"kubernetes.io/projected/681f5ec2-4771-4e6a-b539-8b88971c205e-kube-api-access-rd96s\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518228 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518229 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-daemon-config\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518251 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e82bb4e-d528-49e5-8c0d-02eb57c2e399-tmp-dir\") pod \"node-resolver-6cnbv\" (UID: \"6e82bb4e-d528-49e5-8c0d-02eb57c2e399\") " pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518285 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-modprobe-d\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.519363 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518308 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-systemd-units\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518329 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbg5f\" (UniqueName: \"kubernetes.io/projected/e0cf60e9-379a-4455-b9c5-6c72caa1774e-kube-api-access-gbg5f\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518358 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518382 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-var-lib-cni-multus\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518404 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-hostroot\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518429 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-run-multus-certs\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518452 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ce96221-80e8-4768-a905-818462349deb-kubelet-config\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518489 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-sys\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518502 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ce96221-80e8-4768-a905-818462349deb-kubelet-config\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518566 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-var-lib-cni-multus\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518569 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-host-run-multus-certs\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.518647 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518677 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e82bb4e-d528-49e5-8c0d-02eb57c2e399-tmp-dir\") pod \"node-resolver-6cnbv\" (UID: \"6e82bb4e-d528-49e5-8c0d-02eb57c2e399\") " pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.518698 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:22:20.018679644 +0000 UTC m=+2.950714795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518728 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-var-lib-kubelet\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518747 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-hostroot\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518755 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-node-log\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.520057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518788 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518801 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6h5t\" (UniqueName: \"kubernetes.io/projected/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-kube-api-access-g6h5t\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518808 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-multus-cni-dir\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518837 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3559246-3cfe-44e7-95ea-6a974178f96e-agent-certs\") pod \"konnectivity-agent-qhsdk\" (UID: \"c3559246-3cfe-44e7-95ea-6a974178f96e\") " pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518868 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzjm\" (UniqueName: \"kubernetes.io/projected/c4033dc0-62e3-4974-9d66-c98ea5033f7f-kube-api-access-5gzjm\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518855 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518893 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/262fafeb-4ba5-4b6e-a870-92f41b198b15-host-slash\") pod \"iptables-alerter-7rc59\" (UID: \"262fafeb-4ba5-4b6e-a870-92f41b198b15\") " pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518923 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c86d6b2-647e-4409-9fb3-ca3861e7b9d4-host\") pod \"node-ca-z6ff9\" (UID: \"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4\") " pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518960 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c86d6b2-647e-4409-9fb3-ca3861e7b9d4-host\") pod \"node-ca-z6ff9\" (UID: \"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4\") " pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.518983 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519005 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72c5583a-0e4e-42b6-b0d0-da18faa742ff-cni-binary-copy\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519034 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtjg\" (UniqueName: \"kubernetes.io/projected/72c5583a-0e4e-42b6-b0d0-da18faa742ff-kube-api-access-4gtjg\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519197 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-device-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519230 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-run-systemd\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519246 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0cf60e9-379a-4455-b9c5-6c72caa1774e-ovnkube-config\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519263 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-os-release\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519283 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86fq\" (UniqueName: \"kubernetes.io/projected/262fafeb-4ba5-4b6e-a870-92f41b198b15-kube-api-access-d86fq\") pod \"iptables-alerter-7rc59\" (UID: \"262fafeb-4ba5-4b6e-a870-92f41b198b15\") " pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.520755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519332 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-run-ovn\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.521386 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519351 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-os-release\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.521386 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519363 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c86d6b2-647e-4409-9fb3-ca3861e7b9d4-serviceca\") pod \"node-ca-z6ff9\" (UID: \"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4\") " pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.521386 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519389 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-os-release\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.521386 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519416 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-sysconfig\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.521386 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519440 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-kubelet\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.521386 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519465 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-slash\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.521386 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519486 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72c5583a-0e4e-42b6-b0d0-da18faa742ff-os-release\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.521386 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519531 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72c5583a-0e4e-42b6-b0d0-da18faa742ff-cni-binary-copy\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.521386 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.519856 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c86d6b2-647e-4409-9fb3-ca3861e7b9d4-serviceca\") pod \"node-ca-z6ff9\" (UID: \"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4\") " pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.524300 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.524272 2529 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 05:22:19.527954 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.527933 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlxl\" (UniqueName: \"kubernetes.io/projected/6e82bb4e-d528-49e5-8c0d-02eb57c2e399-kube-api-access-wdlxl\") pod \"node-resolver-6cnbv\" (UID: \"6e82bb4e-d528-49e5-8c0d-02eb57c2e399\") " pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.528072 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.527973 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtjg\" (UniqueName: \"kubernetes.io/projected/72c5583a-0e4e-42b6-b0d0-da18faa742ff-kube-api-access-4gtjg\") pod \"multus-ctmks\" (UID: \"72c5583a-0e4e-42b6-b0d0-da18faa742ff\") " pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.528971 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.528947 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:19.528971 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.528964 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:19.528971 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.528974 2529 projected.go:196] Error preparing data for projected volume kube-api-access-qndvx for pod openshift-network-diagnostics/network-check-target-srm2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:19.529172 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:19.529024 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx podName:f678dcbc-bc33-443a-b1bb-e119d0818a10 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:20.029007873 +0000 UTC m=+2.961043011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qndvx" (UniqueName: "kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx") pod "network-check-target-srm2m" (UID: "f678dcbc-bc33-443a-b1bb-e119d0818a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:19.531378 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.531338 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7wp4\" (UniqueName: \"kubernetes.io/projected/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-kube-api-access-b7wp4\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:19.532544 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.532444 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7w2f\" (UniqueName: \"kubernetes.io/projected/9c86d6b2-647e-4409-9fb3-ca3861e7b9d4-kube-api-access-c7w2f\") pod \"node-ca-z6ff9\" (UID: \"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4\") " pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.533750 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.533728 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6h5t\" (UniqueName: \"kubernetes.io/projected/543e0b9a-76d9-4728-8cdb-3ae735ec3dd7-kube-api-access-g6h5t\") pod \"multus-additional-cni-plugins-xxvrw\" (UID: \"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7\") " pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.561510 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.561483 2529 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:19.620366 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620335 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-systemd\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.620366 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620366 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/681f5ec2-4771-4e6a-b539-8b88971c205e-tmp\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.620595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620383 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-cni-bin\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.620595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620408 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.620595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620463 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-systemd\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.620595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620511 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.620595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620555 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.620595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620558 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-run\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.620595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620595 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-cni-netd\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620603 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-run\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620617 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-host\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620639 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-log-socket\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620647 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-cni-netd\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620675 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-socket-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620697 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-tuned\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620705 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620713 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-host\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620721 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620748 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3559246-3cfe-44e7-95ea-6a974178f96e-konnectivity-ca\") pod \"konnectivity-agent-qhsdk\" (UID: \"c3559246-3cfe-44e7-95ea-6a974178f96e\") " pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620753 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-log-socket\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620770 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-registration-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620793 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-sysctl-conf\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620814 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0cf60e9-379a-4455-b9c5-6c72caa1774e-ovnkube-script-lib\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.620848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620838 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-sys-fs\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620864 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-run-netns\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620865 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-socket-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620887 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-run-openvswitch\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620912 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0cf60e9-379a-4455-b9c5-6c72caa1774e-ovn-node-metrics-cert\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620915 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-registration-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620937 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/262fafeb-4ba5-4b6e-a870-92f41b198b15-iptables-alerter-script\") pod \"iptables-alerter-7rc59\" (UID: \"262fafeb-4ba5-4b6e-a870-92f41b198b15\") " pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620954 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620961 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-sysctl-d\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620596 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-cni-bin\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.620984 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-lib-modules\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621007 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-etc-openvswitch\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621036 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-var-lib-openvswitch\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621066 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621111 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd96s\" (UniqueName: \"kubernetes.io/projected/681f5ec2-4771-4e6a-b539-8b88971c205e-kube-api-access-rd96s\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621149 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-modprobe-d\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621171 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-systemd-units\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.621462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621201 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbg5f\" (UniqueName: \"kubernetes.io/projected/e0cf60e9-379a-4455-b9c5-6c72caa1774e-kube-api-access-gbg5f\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621230 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-sys\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621255 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-var-lib-kubelet\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621282 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-node-log\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621306 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3559246-3cfe-44e7-95ea-6a974178f96e-agent-certs\") pod \"konnectivity-agent-qhsdk\" (UID: \"c3559246-3cfe-44e7-95ea-6a974178f96e\") " pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621331 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzjm\" (UniqueName: \"kubernetes.io/projected/c4033dc0-62e3-4974-9d66-c98ea5033f7f-kube-api-access-5gzjm\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621365 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/262fafeb-4ba5-4b6e-a870-92f41b198b15-host-slash\") pod \"iptables-alerter-7rc59\" (UID: \"262fafeb-4ba5-4b6e-a870-92f41b198b15\") " pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621406 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-device-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621432 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-run-systemd\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621441 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-etc-openvswitch\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621456 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0cf60e9-379a-4455-b9c5-6c72caa1774e-ovnkube-config\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621489 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86fq\" (UniqueName: \"kubernetes.io/projected/262fafeb-4ba5-4b6e-a870-92f41b198b15-kube-api-access-d86fq\") pod \"iptables-alerter-7rc59\" (UID: \"262fafeb-4ba5-4b6e-a870-92f41b198b15\") " pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621502 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3559246-3cfe-44e7-95ea-6a974178f96e-konnectivity-ca\") pod \"konnectivity-agent-qhsdk\" (UID: \"c3559246-3cfe-44e7-95ea-6a974178f96e\") " pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621530 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-run-ovn\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621574 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-sysconfig\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621581 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-run-ovn\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621600 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-sysctl-conf\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.622258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621603 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-kubelet\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621632 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-var-lib-openvswitch\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621640 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-kubelet\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621644 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-slash\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621679 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0cf60e9-379a-4455-b9c5-6c72caa1774e-env-overrides\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621678 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621695 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-sysconfig\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621718 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-kubernetes\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.621813 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-kubernetes\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622186 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-modprobe-d\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622238 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-systemd-units\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622507 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-sys\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622588 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-sys-fs\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622642 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-run-netns\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622687 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-run-openvswitch\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622731 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/262fafeb-4ba5-4b6e-a870-92f41b198b15-host-slash\") pod \"iptables-alerter-7rc59\" (UID: \"262fafeb-4ba5-4b6e-a870-92f41b198b15\") " pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622772 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-var-lib-kubelet\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622778 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0cf60e9-379a-4455-b9c5-6c72caa1774e-env-overrides\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622798 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-node-log\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.622925 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-lib-modules\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.623886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.623080 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0cf60e9-379a-4455-b9c5-6c72caa1774e-ovnkube-script-lib\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.623137 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-run-systemd\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.623147 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-sysctl-d\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.623886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.623167 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/681f5ec2-4771-4e6a-b539-8b88971c205e-tmp\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.623886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.623206 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4033dc0-62e3-4974-9d66-c98ea5033f7f-device-dir\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.623886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.623226 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0cf60e9-379a-4455-b9c5-6c72caa1774e-host-slash\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.623445 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0cf60e9-379a-4455-b9c5-6c72caa1774e-ovnkube-config\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.623886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.623453 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/262fafeb-4ba5-4b6e-a870-92f41b198b15-iptables-alerter-script\") pod \"iptables-alerter-7rc59\" (UID: \"262fafeb-4ba5-4b6e-a870-92f41b198b15\") " pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.624450 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.624425 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/681f5ec2-4771-4e6a-b539-8b88971c205e-etc-tuned\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.624670 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.624647 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0cf60e9-379a-4455-b9c5-6c72caa1774e-ovn-node-metrics-cert\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.625338 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.625322 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3559246-3cfe-44e7-95ea-6a974178f96e-agent-certs\") pod \"konnectivity-agent-qhsdk\" (UID: \"c3559246-3cfe-44e7-95ea-6a974178f96e\") " pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:19.629935 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.629916 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbg5f\" (UniqueName: \"kubernetes.io/projected/e0cf60e9-379a-4455-b9c5-6c72caa1774e-kube-api-access-gbg5f\") pod \"ovnkube-node-gpmnb\" (UID: \"e0cf60e9-379a-4455-b9c5-6c72caa1774e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.630217 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.630196 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd96s\" (UniqueName: \"kubernetes.io/projected/681f5ec2-4771-4e6a-b539-8b88971c205e-kube-api-access-rd96s\") pod \"tuned-4wzrz\" (UID: \"681f5ec2-4771-4e6a-b539-8b88971c205e\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.631214 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.631192 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86fq\" (UniqueName: \"kubernetes.io/projected/262fafeb-4ba5-4b6e-a870-92f41b198b15-kube-api-access-d86fq\") pod \"iptables-alerter-7rc59\" (UID: \"262fafeb-4ba5-4b6e-a870-92f41b198b15\") " pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.631724 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.631704 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzjm\" (UniqueName: \"kubernetes.io/projected/c4033dc0-62e3-4974-9d66-c98ea5033f7f-kube-api-access-5gzjm\") pod \"aws-ebs-csi-driver-node-qvq6l\" (UID: \"c4033dc0-62e3-4974-9d66-c98ea5033f7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:19.701879 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.701740 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ctmks" Jul 28 05:22:19.709268 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.709237 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6cnbv" Jul 28 05:22:19.720977 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.720955 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6ff9" Jul 28 05:22:19.729485 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.729463 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" Jul 28 05:22:19.736533 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.736497 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7rc59" Jul 28 05:22:19.743151 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.743135 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:19.751097 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.751077 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" Jul 28 05:22:19.757626 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.757603 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:19.763397 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:19.763381 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" Jul 28 05:22:20.024434 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.024343 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:20.024434 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.024404 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:20.024668 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:20.024510 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:20.024668 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:20.024592 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:22:21.024572869 +0000 UTC m=+3.956608007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:20.024668 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:20.024509 2529 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:20.024828 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:20.024692 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret podName:7ce96221-80e8-4768-a905-818462349deb nodeName:}" failed. No retries permitted until 2026-07-28 05:22:21.024677458 +0000 UTC m=+3.956712599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret") pod "global-pull-secret-syncer-r52w4" (UID: "7ce96221-80e8-4768-a905-818462349deb") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:20.121160 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.121129 2529 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:20.125022 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.125002 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:20.125221 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:20.125201 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:20.125265 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:20.125227 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:20.125265 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:20.125237 2529 projected.go:196] Error preparing data for projected volume kube-api-access-qndvx for pod openshift-network-diagnostics/network-check-target-srm2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:20.125341 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:20.125285 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx podName:f678dcbc-bc33-443a-b1bb-e119d0818a10 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:21.125270106 +0000 UTC m=+4.057305245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qndvx" (UniqueName: "kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx") pod "network-check-target-srm2m" (UID: "f678dcbc-bc33-443a-b1bb-e119d0818a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:20.385254 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:20.385229 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543e0b9a_76d9_4728_8cdb_3ae735ec3dd7.slice/crio-dfddd3670f3412fa3e90a977a6ab66da05100ceb181ec77c769e3ffd29373ba5 WatchSource:0}: Error finding container dfddd3670f3412fa3e90a977a6ab66da05100ceb181ec77c769e3ffd29373ba5: Status 404 returned error can't find the container with id dfddd3670f3412fa3e90a977a6ab66da05100ceb181ec77c769e3ffd29373ba5 Jul 28 05:22:20.386880 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:20.386849 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c5583a_0e4e_42b6_b0d0_da18faa742ff.slice/crio-e97c57f4838dde7a57d521c03b22e609ab769d06080792827a71854cbe0be2ce WatchSource:0}: Error finding container e97c57f4838dde7a57d521c03b22e609ab769d06080792827a71854cbe0be2ce: Status 404 returned error can't find the container with id e97c57f4838dde7a57d521c03b22e609ab769d06080792827a71854cbe0be2ce Jul 28 05:22:20.387566 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:20.387423 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e82bb4e_d528_49e5_8c0d_02eb57c2e399.slice/crio-c586f092f9d23c486675f495ca315670c599be47253b700cfd543dc6634c669e WatchSource:0}: Error finding container c586f092f9d23c486675f495ca315670c599be47253b700cfd543dc6634c669e: Status 404 returned error can't find the container with id c586f092f9d23c486675f495ca315670c599be47253b700cfd543dc6634c669e Jul 28 05:22:20.388185 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:20.388148 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cf60e9_379a_4455_b9c5_6c72caa1774e.slice/crio-1b9ce356969364ab376c3db257aff4effdd245bcdb95e301d49acece5c08671b WatchSource:0}: Error finding container 1b9ce356969364ab376c3db257aff4effdd245bcdb95e301d49acece5c08671b: Status 404 returned error can't find the container with id 1b9ce356969364ab376c3db257aff4effdd245bcdb95e301d49acece5c08671b Jul 28 05:22:20.389343 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:20.389303 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681f5ec2_4771_4e6a_b539_8b88971c205e.slice/crio-b75d9628494bb02c2361f5bdc956d6533747ae1d9e54613fb71fb4ec5fc10511 WatchSource:0}: Error finding container b75d9628494bb02c2361f5bdc956d6533747ae1d9e54613fb71fb4ec5fc10511: Status 404 returned error can't find the container with id b75d9628494bb02c2361f5bdc956d6533747ae1d9e54613fb71fb4ec5fc10511 Jul 28 05:22:20.416540 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:20.416232 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3559246_3cfe_44e7_95ea_6a974178f96e.slice/crio-77257c3e23136d019b7868d1e77b422b56d233a6cfe30924e4a83557d279f844 WatchSource:0}: Error finding container 77257c3e23136d019b7868d1e77b422b56d233a6cfe30924e4a83557d279f844: Status 404 returned error can't find the container with id 77257c3e23136d019b7868d1e77b422b56d233a6cfe30924e4a83557d279f844 Jul 28 05:22:20.418688 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:20.418670 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86d6b2_647e_4409_9fb3_ca3861e7b9d4.slice/crio-9df242720c4f177720dc1688e73a541150ef2c78b8a3b4fa0f3e2739a936a041 WatchSource:0}: Error finding container 9df242720c4f177720dc1688e73a541150ef2c78b8a3b4fa0f3e2739a936a041: Status 404 returned error can't find the container with id 9df242720c4f177720dc1688e73a541150ef2c78b8a3b4fa0f3e2739a936a041 Jul 28 05:22:20.446383 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.446354 2529 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 05:17:19 +0000 UTC" deadline="2028-01-14 20:43:56.970800362 +0000 UTC" Jul 28 05:22:20.446383 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.446384 2529 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12855h21m36.524420695s" Jul 28 05:22:20.580008 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.579860 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:20.580114 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:20.580088 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:20.587147 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.587120 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6cnbv" event={"ID":"6e82bb4e-d528-49e5-8c0d-02eb57c2e399","Type":"ContainerStarted","Data":"c586f092f9d23c486675f495ca315670c599be47253b700cfd543dc6634c669e"} Jul 28 05:22:20.588535 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.588485 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" event={"ID":"681f5ec2-4771-4e6a-b539-8b88971c205e","Type":"ContainerStarted","Data":"b75d9628494bb02c2361f5bdc956d6533747ae1d9e54613fb71fb4ec5fc10511"} Jul 28 05:22:20.589622 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.589598 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" event={"ID":"e0cf60e9-379a-4455-b9c5-6c72caa1774e","Type":"ContainerStarted","Data":"1b9ce356969364ab376c3db257aff4effdd245bcdb95e301d49acece5c08671b"} Jul 28 05:22:20.594323 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.594295 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" event={"ID":"6d68380c15b9e00b00440ef022ddd164","Type":"ContainerStarted","Data":"9e07afba2338763923dc99c23fda738f24d8af6f567a69d07b8acbd423c131b5"} Jul 28 05:22:20.595470 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.595444 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6ff9" event={"ID":"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4","Type":"ContainerStarted","Data":"9df242720c4f177720dc1688e73a541150ef2c78b8a3b4fa0f3e2739a936a041"} Jul 28 05:22:20.596773 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.596752 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhsdk" event={"ID":"c3559246-3cfe-44e7-95ea-6a974178f96e","Type":"ContainerStarted","Data":"77257c3e23136d019b7868d1e77b422b56d233a6cfe30924e4a83557d279f844"} Jul 28 05:22:20.597756 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.597736 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ctmks" event={"ID":"72c5583a-0e4e-42b6-b0d0-da18faa742ff","Type":"ContainerStarted","Data":"e97c57f4838dde7a57d521c03b22e609ab769d06080792827a71854cbe0be2ce"} Jul 28 05:22:20.598719 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.598695 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" event={"ID":"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7","Type":"ContainerStarted","Data":"dfddd3670f3412fa3e90a977a6ab66da05100ceb181ec77c769e3ffd29373ba5"} Jul 28 05:22:20.599664 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.599648 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" event={"ID":"c4033dc0-62e3-4974-9d66-c98ea5033f7f","Type":"ContainerStarted","Data":"76a7b1057c062aa2272d8ae7ec2d9ffc3d4d95f9b0b38342ac3e8034ed6cf28d"} Jul 28 05:22:20.600567 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.600548 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7rc59" event={"ID":"262fafeb-4ba5-4b6e-a870-92f41b198b15","Type":"ContainerStarted","Data":"15748e6619e315ff1effe1c4da4dc26d841eaae5ac6fece8c85cc56a490805f5"} Jul 28 05:22:20.607984 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:20.607945 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" podStartSLOduration=2.607933932 podStartE2EDuration="2.607933932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:20.607116122 +0000 UTC m=+3.539151284" watchObservedRunningTime="2026-07-28 05:22:20.607933932 +0000 UTC m=+3.539969091" Jul 28 05:22:21.033130 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:21.032316 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:21.033130 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:21.032433 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:21.033130 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.032600 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:21.033130 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.032664 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:22:23.032644159 +0000 UTC m=+5.964679303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:21.033130 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.033046 2529 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:21.033130 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.033094 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret podName:7ce96221-80e8-4768-a905-818462349deb nodeName:}" failed. No retries permitted until 2026-07-28 05:22:23.033080498 +0000 UTC m=+5.965115641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret") pod "global-pull-secret-syncer-r52w4" (UID: "7ce96221-80e8-4768-a905-818462349deb") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:21.136830 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:21.133548 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:21.136830 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.133754 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:21.136830 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.133772 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:21.136830 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.133785 2529 projected.go:196] Error preparing data for projected volume kube-api-access-qndvx for pod openshift-network-diagnostics/network-check-target-srm2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:21.136830 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.133840 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx podName:f678dcbc-bc33-443a-b1bb-e119d0818a10 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:23.133821888 +0000 UTC m=+6.065857031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qndvx" (UniqueName: "kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx") pod "network-check-target-srm2m" (UID: "f678dcbc-bc33-443a-b1bb-e119d0818a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:21.446823 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:21.446715 2529 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 05:17:19 +0000 UTC" deadline="2028-03-23 18:58:29.659170209 +0000 UTC" Jul 28 05:22:21.446823 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:21.446774 2529 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14509h36m8.212400729s" Jul 28 05:22:21.580554 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:21.580069 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:21.580554 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.580228 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:21.581057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:21.580913 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:21.581057 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:21.581018 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:21.616085 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:21.616044 2529 generic.go:356] "Generic (PLEG): container finished" podID="67a91ffb87b056369246a9445ebaa0cf" containerID="a6360dca4ff24be87688b7ab58ee13d9647798c254b3705dc420b78727fd6a87" exitCode=0 Jul 28 05:22:21.616275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:21.616211 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" event={"ID":"67a91ffb87b056369246a9445ebaa0cf","Type":"ContainerDied","Data":"a6360dca4ff24be87688b7ab58ee13d9647798c254b3705dc420b78727fd6a87"} Jul 28 05:22:22.579925 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:22.579842 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:22.580388 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:22.579985 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:22.648882 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:22.648078 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" event={"ID":"67a91ffb87b056369246a9445ebaa0cf","Type":"ContainerStarted","Data":"77db46c6329742e458f7889bec010a68c48e930bf09b2be1e836eeab745b8cf1"} Jul 28 05:22:23.049549 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:23.049449 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:23.049549 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:23.049536 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:23.049798 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.049686 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:23.049798 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.049748 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:22:27.049728777 +0000 UTC m=+9.981763919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:23.050143 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.050123 2529 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:23.050301 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.050180 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret podName:7ce96221-80e8-4768-a905-818462349deb nodeName:}" failed. No retries permitted until 2026-07-28 05:22:27.050163361 +0000 UTC m=+9.982198514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret") pod "global-pull-secret-syncer-r52w4" (UID: "7ce96221-80e8-4768-a905-818462349deb") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:23.150568 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:23.150528 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:23.150782 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.150759 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:23.150872 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.150788 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:23.150872 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.150803 2529 projected.go:196] Error preparing data for projected volume kube-api-access-qndvx for pod openshift-network-diagnostics/network-check-target-srm2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:23.150981 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.150894 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx podName:f678dcbc-bc33-443a-b1bb-e119d0818a10 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:27.150872727 +0000 UTC m=+10.082907968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qndvx" (UniqueName: "kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx") pod "network-check-target-srm2m" (UID: "f678dcbc-bc33-443a-b1bb-e119d0818a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:23.582100 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:23.581621 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:23.582100 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:23.581637 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:23.582100 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.581765 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:23.582100 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:23.581860 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:24.580091 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:24.580054 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:24.580272 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:24.580208 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:25.581274 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:25.580646 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:25.581274 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:25.580809 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:25.581274 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:25.580853 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:25.581274 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:25.580925 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:26.580165 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:26.579818 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:26.580165 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:26.579967 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:27.085021 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:27.084989 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:27.085463 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:27.085057 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:27.085463 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.085121 2529 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:27.085463 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.085167 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:27.085463 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.085212 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret podName:7ce96221-80e8-4768-a905-818462349deb nodeName:}" failed. No retries permitted until 2026-07-28 05:22:35.085188296 +0000 UTC m=+18.017223446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret") pod "global-pull-secret-syncer-r52w4" (UID: "7ce96221-80e8-4768-a905-818462349deb") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:27.085463 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.085232 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:22:35.085221728 +0000 UTC m=+18.017256875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:27.186297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:27.186232 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:27.186463 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.186422 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:27.186463 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.186444 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:27.186463 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.186455 2529 projected.go:196] Error preparing data for projected volume kube-api-access-qndvx for pod openshift-network-diagnostics/network-check-target-srm2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:27.186673 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.186529 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx podName:f678dcbc-bc33-443a-b1bb-e119d0818a10 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:35.186493211 +0000 UTC m=+18.118528373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qndvx" (UniqueName: "kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx") pod "network-check-target-srm2m" (UID: "f678dcbc-bc33-443a-b1bb-e119d0818a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:27.581689 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:27.581616 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:27.581874 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.581755 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:27.582184 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:27.582165 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:27.582288 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:27.582265 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:28.580807 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:28.580773 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:28.581204 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:28.580899 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:29.580500 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:29.580463 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:29.580687 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:29.580464 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:29.580687 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:29.580612 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:29.580801 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:29.580728 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:30.580789 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:30.580750 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:30.581250 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:30.580883 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:31.580629 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:31.580591 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:31.580808 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:31.580598 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:31.580808 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:31.580713 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:31.581176 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:31.580803 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:32.580813 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:32.580771 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:32.581262 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:32.580904 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:33.580313 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:33.580275 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:33.580533 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:33.580423 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:33.580533 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:33.580480 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:33.580663 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:33.580605 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:34.579945 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:34.579905 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:34.580457 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:34.580020 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:35.142342 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:35.142299 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:35.142554 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:35.142373 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:35.142554 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.142444 2529 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:35.142554 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.142491 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:35.142554 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.142533 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret podName:7ce96221-80e8-4768-a905-818462349deb nodeName:}" failed. No retries permitted until 2026-07-28 05:22:51.142502762 +0000 UTC m=+34.074537900 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret") pod "global-pull-secret-syncer-r52w4" (UID: "7ce96221-80e8-4768-a905-818462349deb") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:35.142554 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.142553 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:22:51.142545225 +0000 UTC m=+34.074580364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:35.243384 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:35.243345 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:35.243571 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.243508 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:35.243571 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.243542 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:35.243571 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.243552 2529 projected.go:196] Error preparing data for projected volume kube-api-access-qndvx for pod openshift-network-diagnostics/network-check-target-srm2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:35.243730 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.243614 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx podName:f678dcbc-bc33-443a-b1bb-e119d0818a10 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:51.243596135 +0000 UTC m=+34.175631275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qndvx" (UniqueName: "kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx") pod "network-check-target-srm2m" (UID: "f678dcbc-bc33-443a-b1bb-e119d0818a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:35.579952 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:35.579864 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:35.580104 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:35.579866 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:35.580104 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.579994 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:35.580507 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:35.580108 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:36.580711 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:36.580673 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:36.581118 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:36.580816 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:37.581073 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:37.581036 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:37.581508 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:37.581184 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:37.581508 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:37.581220 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:37.581508 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:37.581344 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:38.580145 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:38.580106 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:38.580319 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:38.580254 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:39.580229 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.580033 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:39.580876 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.580098 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:39.580876 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:39.580307 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:39.580876 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:39.580379 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:39.683108 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.683077 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ctmks" event={"ID":"72c5583a-0e4e-42b6-b0d0-da18faa742ff","Type":"ContainerStarted","Data":"623f1c9ecf48b364ac3adac044143213e8e188531a3e4436af0719593a4440e6"} Jul 28 05:22:39.684629 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.684607 2529 generic.go:356] "Generic (PLEG): container finished" podID="543e0b9a-76d9-4728-8cdb-3ae735ec3dd7" containerID="0b0ee430c6e9b0e84ef3d5d1cdb771772313669c0dc96d87ba287db043e3dce5" exitCode=0 Jul 28 05:22:39.684717 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.684690 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" event={"ID":"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7","Type":"ContainerDied","Data":"0b0ee430c6e9b0e84ef3d5d1cdb771772313669c0dc96d87ba287db043e3dce5"} Jul 28 05:22:39.686166 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.686144 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" event={"ID":"c4033dc0-62e3-4974-9d66-c98ea5033f7f","Type":"ContainerStarted","Data":"160c79eb7119edc3b5ab2c44dc89708ab96ec0900d0f24828915d4e3e690fabc"} Jul 28 05:22:39.687606 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.687491 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6cnbv" event={"ID":"6e82bb4e-d528-49e5-8c0d-02eb57c2e399","Type":"ContainerStarted","Data":"2cde15fd0c0a943a7a27540b237bb028d994cc4569736981528fad6848fc4247"} Jul 28 05:22:39.688993 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.688969 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" event={"ID":"681f5ec2-4771-4e6a-b539-8b88971c205e","Type":"ContainerStarted","Data":"a57ea199b146778bf09eb150f72b16f5e04e10fa6b71e519f9b8590847bb4b79"} Jul 28 05:22:39.691647 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.691623 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" event={"ID":"e0cf60e9-379a-4455-b9c5-6c72caa1774e","Type":"ContainerStarted","Data":"eb1f434b3eaf2d3337234a5be606fe8a0e74eef303735a0dbe5705981b4f7413"} Jul 28 05:22:39.691747 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.691722 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" event={"ID":"e0cf60e9-379a-4455-b9c5-6c72caa1774e","Type":"ContainerStarted","Data":"830bdd73530a9eb01a62a4afef354fcdfcfef0a894d08d964701c971d9de7468"} Jul 28 05:22:39.691817 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.691767 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" event={"ID":"e0cf60e9-379a-4455-b9c5-6c72caa1774e","Type":"ContainerStarted","Data":"70645a0123eed7d788110aa7ceb39a4e0ad85e4ecf1485c0431a6f51d719fc76"} Jul 28 05:22:39.691869 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.691815 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" event={"ID":"e0cf60e9-379a-4455-b9c5-6c72caa1774e","Type":"ContainerStarted","Data":"e3fc2dfef6c376035804d11060685b22f7b121d50a7db8fcadb746136c046217"} Jul 28 05:22:39.691907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.691866 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" event={"ID":"e0cf60e9-379a-4455-b9c5-6c72caa1774e","Type":"ContainerStarted","Data":"09dd47b992f28e62696311374c1fcb67f295bb772d2f8d457b189cdf64dfc078"} Jul 28 05:22:39.693012 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.692981 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6ff9" event={"ID":"9c86d6b2-647e-4409-9fb3-ca3861e7b9d4","Type":"ContainerStarted","Data":"a3958faa22658f172ada5b0fe8ff79722229dbacd50968cb4212e29674385878"} Jul 28 05:22:39.694297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.694274 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhsdk" event={"ID":"c3559246-3cfe-44e7-95ea-6a974178f96e","Type":"ContainerStarted","Data":"3435c4788a6e7fdd8b3ccce1aed3bc9eeccf9b2bf69430721274ae67b2c37fd4"} Jul 28 05:22:39.697734 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.697694 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" podStartSLOduration=21.697683484 podStartE2EDuration="21.697683484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:22.662611481 +0000 UTC m=+5.594646642" watchObservedRunningTime="2026-07-28 05:22:39.697683484 +0000 UTC m=+22.629718669" Jul 28 05:22:39.697984 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.697958 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ctmks" podStartSLOduration=4.017071649 podStartE2EDuration="22.697952236s" totalImagesPullingTime="18.680880587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:39.69757937 +0000 UTC m=+22.629614530" watchObservedRunningTime="2026-07-28 05:22:39.697952236 +0000 UTC m=+22.629987396" Jul 28 05:22:39.724757 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.724713 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qhsdk" podStartSLOduration=8.378581364 podStartE2EDuration="22.724696789s" totalImagesPullingTime="14.346115425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:39.7126143 +0000 UTC m=+22.644649459" watchObservedRunningTime="2026-07-28 05:22:39.724696789 +0000 UTC m=+22.656731949" Jul 28 05:22:39.724916 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.724889 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6cnbv" podStartSLOduration=4.132150551 podStartE2EDuration="22.724878693s" totalImagesPullingTime="18.592728142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:39.724544944 +0000 UTC m=+22.656580104" watchObservedRunningTime="2026-07-28 05:22:39.724878693 +0000 UTC m=+22.656913854" Jul 28 05:22:39.736974 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.736921 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z6ff9" podStartSLOduration=4.153394533 podStartE2EDuration="22.736905708s" totalImagesPullingTime="18.583511175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:39.736350224 +0000 UTC m=+22.668385385" watchObservedRunningTime="2026-07-28 05:22:39.736905708 +0000 UTC m=+22.668940889" Jul 28 05:22:39.780582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:39.780531 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4wzrz" podStartSLOduration=4.187419938 podStartE2EDuration="22.780498477s" totalImagesPullingTime="18.593078539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:39.779889872 +0000 UTC m=+22.711925033" watchObservedRunningTime="2026-07-28 05:22:39.780498477 +0000 UTC m=+22.712533636" Jul 28 05:22:40.580304 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:40.580269 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:40.580753 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:40.580392 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:40.622106 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:40.622073 2529 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 05:22:40.701782 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:40.701744 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" event={"ID":"e0cf60e9-379a-4455-b9c5-6c72caa1774e","Type":"ContainerStarted","Data":"a1dcc384c3252c1284b4fab4b5c23529bf4533442799432cc45b9d232625ee3a"} Jul 28 05:22:40.703755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:40.703724 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" event={"ID":"c4033dc0-62e3-4974-9d66-c98ea5033f7f","Type":"ContainerStarted","Data":"e67e0d5ada1df8df7c4d4a1244cbdf3f296b0e7afb2c47c7a78229e34e80ce71"} Jul 28 05:22:40.705343 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:40.705319 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7rc59" event={"ID":"262fafeb-4ba5-4b6e-a870-92f41b198b15","Type":"ContainerStarted","Data":"169f9d4533424d0699f300ac35fd1a4f99f60603218718691d450b6372894476"} Jul 28 05:22:40.718930 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:40.718882 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7rc59" podStartSLOduration=5.132238841 podStartE2EDuration="23.718867179s" totalImagesPullingTime="18.586628338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:40.718702381 +0000 UTC m=+23.650737541" watchObservedRunningTime="2026-07-28 05:22:40.718867179 +0000 UTC m=+23.650902361" Jul 28 05:22:41.472979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:41.472869 2529 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T05:22:40.622103773Z","UUID":"d83a45fd-e456-4068-9917-01ede1e69d55","Handler":null,"Name":"","Endpoint":""} Jul 28 05:22:41.474643 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:41.474619 2529 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 05:22:41.474756 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:41.474680 2529 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 05:22:41.580511 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:41.580484 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:41.580973 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:41.580488 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:41.580973 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:41.580638 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:41.580973 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:41.580715 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:42.579900 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:42.579809 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:42.580036 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:42.579939 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:42.714275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:42.713463 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" event={"ID":"c4033dc0-62e3-4974-9d66-c98ea5033f7f","Type":"ContainerStarted","Data":"dcb2cb7eeea14994d9d65791bc9f838a09bfc953d580de8b47b5e66b54027e7e"} Jul 28 05:22:42.718165 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:42.718137 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" event={"ID":"e0cf60e9-379a-4455-b9c5-6c72caa1774e","Type":"ContainerStarted","Data":"469aa2c26aad2c59bf7fce05a9ff5a8bf4d567cb2b5b7fbf8e911c2351696d21"} Jul 28 05:22:42.728678 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:42.728633 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvq6l" podStartSLOduration=4.458512729 podStartE2EDuration="25.728618476s" totalImagesPullingTime="21.270105747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:42.728563751 +0000 UTC m=+25.660598913" watchObservedRunningTime="2026-07-28 05:22:42.728618476 +0000 UTC m=+25.660653636" Jul 28 05:22:43.580615 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:43.580402 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:43.580804 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:43.580451 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:43.580804 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:43.580734 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:43.580804 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:43.580777 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:44.276250 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.276153 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:44.278666 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.276736 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:44.580195 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.580114 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:44.580318 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:44.580225 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:44.726578 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.726542 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" event={"ID":"e0cf60e9-379a-4455-b9c5-6c72caa1774e","Type":"ContainerStarted","Data":"a9f7cf57e353e66aa3bfc4ee58d78dc54b143101b419dbbf923b15dd7660fdb2"} Jul 28 05:22:44.726769 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.726749 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:44.726822 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.726813 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:44.728499 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.728473 2529 generic.go:356] "Generic (PLEG): container finished" podID="543e0b9a-76d9-4728-8cdb-3ae735ec3dd7" containerID="e217e52ccfb9b577eec2ea24aad880dbd7102b7c9da65bf5b453828241878f82" exitCode=0 Jul 28 05:22:44.728642 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.728556 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" event={"ID":"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7","Type":"ContainerDied","Data":"e217e52ccfb9b577eec2ea24aad880dbd7102b7c9da65bf5b453828241878f82"} Jul 28 05:22:44.743939 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.743787 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:44.753621 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.753584 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" podStartSLOduration=9.06502232 podStartE2EDuration="27.753571993s" totalImagesPullingTime="18.688549673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:44.752434804 +0000 UTC m=+27.684469964" watchObservedRunningTime="2026-07-28 05:22:44.753571993 +0000 UTC m=+27.685607147" Jul 28 05:22:44.795714 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.795685 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:44.873031 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:44.873001 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:45.579904 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:45.579864 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:45.580364 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:45.579871 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:45.580364 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:45.580036 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:45.580364 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:45.580082 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:45.730654 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:45.730620 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:45.745452 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:45.745423 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:22:46.145724 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.145642 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r52w4"] Jul 28 05:22:46.145879 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.145773 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:46.145929 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:46.145869 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:46.148623 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.148586 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-srm2m"] Jul 28 05:22:46.148773 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.148695 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:46.148854 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:46.148792 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:46.149206 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.149187 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qntsm"] Jul 28 05:22:46.149275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.149263 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:46.149372 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:46.149347 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:46.286954 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.286911 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:46.388461 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.388427 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:46.734507 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.734474 2529 generic.go:356] "Generic (PLEG): container finished" podID="543e0b9a-76d9-4728-8cdb-3ae735ec3dd7" containerID="883a8ef365b9b8a9823b089ac4cd5623f1008f56a8cea423015a1b98accc0e38" exitCode=0 Jul 28 05:22:46.734979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:46.734559 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" event={"ID":"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7","Type":"ContainerDied","Data":"883a8ef365b9b8a9823b089ac4cd5623f1008f56a8cea423015a1b98accc0e38"} Jul 28 05:22:47.319216 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:47.319187 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:47.369080 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:47.369018 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:47.580688 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:47.580618 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:47.580845 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:47.580706 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:47.580845 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:47.580728 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:47.580845 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:47.580704 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:47.580845 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:47.580805 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:47.581014 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:47.580880 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:48.744496 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:48.744468 2529 generic.go:356] "Generic (PLEG): container finished" podID="543e0b9a-76d9-4728-8cdb-3ae735ec3dd7" containerID="fa90867ef77cf87ff814a7201c33210c78779230eb21bf35048496137b2709c4" exitCode=0 Jul 28 05:22:48.745073 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:48.744563 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" event={"ID":"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7","Type":"ContainerDied","Data":"fa90867ef77cf87ff814a7201c33210c78779230eb21bf35048496137b2709c4"} Jul 28 05:22:49.184410 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:49.184379 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:49.285628 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:49.285592 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:49.364848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:49.364815 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:49.365017 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:49.364960 2529 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 28 05:22:49.366218 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:49.366191 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhsdk" Jul 28 05:22:49.423507 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:49.423467 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:49.580811 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:49.580723 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:49.580811 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:49.580768 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:49.581014 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:49.580733 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:49.581014 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:49.580866 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:49.581014 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:49.580950 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:49.581133 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:49.581053 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:51.163417 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:51.163385 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:51.164012 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:51.163467 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:51.164012 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.163562 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:51.164012 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.163578 2529 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:51.164012 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.163625 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret podName:7ce96221-80e8-4768-a905-818462349deb nodeName:}" failed. No retries permitted until 2026-07-28 05:23:23.163611837 +0000 UTC m=+66.095646976 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret") pod "global-pull-secret-syncer-r52w4" (UID: "7ce96221-80e8-4768-a905-818462349deb") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:51.164012 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.163645 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:23:23.16363533 +0000 UTC m=+66.095670474 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:51.263985 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:51.263950 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:51.264137 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.264078 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:51.264137 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.264093 2529 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:51.264137 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.264101 2529 projected.go:196] Error preparing data for projected volume kube-api-access-qndvx for pod openshift-network-diagnostics/network-check-target-srm2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:51.264351 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.264158 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx podName:f678dcbc-bc33-443a-b1bb-e119d0818a10 nodeName:}" failed. No retries permitted until 2026-07-28 05:23:23.264136178 +0000 UTC m=+66.196171332 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qndvx" (UniqueName: "kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx") pod "network-check-target-srm2m" (UID: "f678dcbc-bc33-443a-b1bb-e119d0818a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:51.584296 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:51.584206 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:51.584562 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:51.584308 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:51.584562 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.584320 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r52w4" podUID="7ce96221-80e8-4768-a905-818462349deb" Jul 28 05:22:51.584562 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:51.584336 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:51.584562 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.584416 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-srm2m" podUID="f678dcbc-bc33-443a-b1bb-e119d0818a10" Jul 28 05:22:51.584562 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:51.584503 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:22:52.861050 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.861023 2529 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeReady" Jul 28 05:22:52.861643 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.861190 2529 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 05:22:52.900659 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.900629 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n7tqp"] Jul 28 05:22:52.904714 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.904692 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-87bqg"] Jul 28 05:22:52.904845 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.904827 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:52.909132 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.907720 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87bqg" Jul 28 05:22:52.909132 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.907743 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cs9jc\"" Jul 28 05:22:52.909132 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.907955 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 05:22:52.909132 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.908437 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 05:22:52.909996 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.909812 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 05:22:52.910237 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.910210 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 05:22:52.910325 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.910270 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qnq98\"" Jul 28 05:22:52.910325 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.910303 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 05:22:52.912569 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.912488 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n7tqp"] Jul 28 05:22:52.916426 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.915579 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87bqg"] Jul 28 05:22:52.975385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.975360 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:52.975505 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.975400 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0855a53-29d0-4bad-95a4-e0696e3357c4-tmp-dir\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:52.975587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.975499 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqb8q\" (UniqueName: \"kubernetes.io/projected/c0855a53-29d0-4bad-95a4-e0696e3357c4-kube-api-access-hqb8q\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:52.975638 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.975601 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c0855a53-29d0-4bad-95a4-e0696e3357c4-config-volume\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:52.975680 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.975654 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hszxt\" (UniqueName: \"kubernetes.io/projected/f4600925-15b9-4d3e-8329-dfa4d59fe6f3-kube-api-access-hszxt\") pod \"ingress-canary-87bqg\" (UID: \"f4600925-15b9-4d3e-8329-dfa4d59fe6f3\") " pod="openshift-ingress-canary/ingress-canary-87bqg" Jul 28 05:22:52.975762 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:52.975744 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4600925-15b9-4d3e-8329-dfa4d59fe6f3-cert\") pod \"ingress-canary-87bqg\" (UID: \"f4600925-15b9-4d3e-8329-dfa4d59fe6f3\") " pod="openshift-ingress-canary/ingress-canary-87bqg" Jul 28 05:22:53.009618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.009591 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:53.059227 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.059202 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:53.076093 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.076066 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c0855a53-29d0-4bad-95a4-e0696e3357c4-config-volume\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:53.076194 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.076123 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hszxt\" (UniqueName: \"kubernetes.io/projected/f4600925-15b9-4d3e-8329-dfa4d59fe6f3-kube-api-access-hszxt\") pod \"ingress-canary-87bqg\" (UID: \"f4600925-15b9-4d3e-8329-dfa4d59fe6f3\") " pod="openshift-ingress-canary/ingress-canary-87bqg" Jul 28 05:22:53.076252 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.076217 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4600925-15b9-4d3e-8329-dfa4d59fe6f3-cert\") pod \"ingress-canary-87bqg\" (UID: \"f4600925-15b9-4d3e-8329-dfa4d59fe6f3\") " pod="openshift-ingress-canary/ingress-canary-87bqg" Jul 28 05:22:53.076298 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.076251 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:53.076298 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.076275 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0855a53-29d0-4bad-95a4-e0696e3357c4-tmp-dir\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:53.076387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.076318 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqb8q\" (UniqueName: \"kubernetes.io/projected/c0855a53-29d0-4bad-95a4-e0696e3357c4-kube-api-access-hqb8q\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:53.076387 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:53.076362 2529 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:22:53.076477 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:53.076439 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls podName:c0855a53-29d0-4bad-95a4-e0696e3357c4 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:53.576423163 +0000 UTC m=+36.508458325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls") pod "dns-default-n7tqp" (UID: "c0855a53-29d0-4bad-95a4-e0696e3357c4") : secret "dns-default-metrics-tls" not found Jul 28 05:22:53.076661 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.076641 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0855a53-29d0-4bad-95a4-e0696e3357c4-tmp-dir\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:53.076821 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.076807 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c0855a53-29d0-4bad-95a4-e0696e3357c4-config-volume\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:53.080094 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.080075 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4600925-15b9-4d3e-8329-dfa4d59fe6f3-cert\") pod \"ingress-canary-87bqg\" (UID: \"f4600925-15b9-4d3e-8329-dfa4d59fe6f3\") " pod="openshift-ingress-canary/ingress-canary-87bqg" Jul 28 05:22:53.083757 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.083732 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqb8q\" (UniqueName: \"kubernetes.io/projected/c0855a53-29d0-4bad-95a4-e0696e3357c4-kube-api-access-hqb8q\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:53.084144 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.084119 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hszxt\" (UniqueName: \"kubernetes.io/projected/f4600925-15b9-4d3e-8329-dfa4d59fe6f3-kube-api-access-hszxt\") pod \"ingress-canary-87bqg\" (UID: \"f4600925-15b9-4d3e-8329-dfa4d59fe6f3\") " pod="openshift-ingress-canary/ingress-canary-87bqg" Jul 28 05:22:53.227610 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.227494 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87bqg" Jul 28 05:22:53.354373 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.354214 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87bqg"] Jul 28 05:22:53.357949 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:22:53.357924 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4600925_15b9_4d3e_8329_dfa4d59fe6f3.slice/crio-d7c7035660659b7cf0b2075acd53571d12c776cf421c07b8d70cef0b078b584f WatchSource:0}: Error finding container d7c7035660659b7cf0b2075acd53571d12c776cf421c07b8d70cef0b078b584f: Status 404 returned error can't find the container with id d7c7035660659b7cf0b2075acd53571d12c776cf421c07b8d70cef0b078b584f Jul 28 05:22:53.580110 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.579980 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:22:53.580110 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.580015 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:53.580110 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.579989 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:22:53.580110 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.580104 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:22:53.580439 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:53.580152 2529 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:22:53.580439 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:53.580207 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls podName:c0855a53-29d0-4bad-95a4-e0696e3357c4 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:54.580190779 +0000 UTC m=+37.512225920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls") pod "dns-default-n7tqp" (UID: "c0855a53-29d0-4bad-95a4-e0696e3357c4") : secret "dns-default-metrics-tls" not found Jul 28 05:22:53.582555 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.582510 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 05:22:53.582699 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.582685 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 05:22:53.583624 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.583601 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 05:22:53.583624 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.583604 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqsrp\"" Jul 28 05:22:53.583798 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.583655 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 05:22:53.583798 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.583678 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mr6ww\"" Jul 28 05:22:53.757497 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.757470 2529 generic.go:356] "Generic (PLEG): container finished" podID="543e0b9a-76d9-4728-8cdb-3ae735ec3dd7" containerID="f0f446b1b4fa84254aa238dcd5b8c2f7b7109520427c2d53714cdf74ff60c2cf" exitCode=0 Jul 28 05:22:53.757720 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.757554 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" event={"ID":"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7","Type":"ContainerDied","Data":"f0f446b1b4fa84254aa238dcd5b8c2f7b7109520427c2d53714cdf74ff60c2cf"} Jul 28 05:22:53.758731 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:53.758707 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87bqg" event={"ID":"f4600925-15b9-4d3e-8329-dfa4d59fe6f3","Type":"ContainerStarted","Data":"d7c7035660659b7cf0b2075acd53571d12c776cf421c07b8d70cef0b078b584f"} Jul 28 05:22:54.589027 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:54.588988 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:54.589492 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:54.589178 2529 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:22:54.589492 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:54.589262 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls podName:c0855a53-29d0-4bad-95a4-e0696e3357c4 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:56.58924023 +0000 UTC m=+39.521275378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls") pod "dns-default-n7tqp" (UID: "c0855a53-29d0-4bad-95a4-e0696e3357c4") : secret "dns-default-metrics-tls" not found Jul 28 05:22:54.765161 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:54.765126 2529 generic.go:356] "Generic (PLEG): container finished" podID="543e0b9a-76d9-4728-8cdb-3ae735ec3dd7" containerID="e192f9782ecf2b32b33247b24ed7332bb08ba3d355700a6a681c0da8ab196769" exitCode=0 Jul 28 05:22:54.765320 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:54.765216 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" event={"ID":"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7","Type":"ContainerDied","Data":"e192f9782ecf2b32b33247b24ed7332bb08ba3d355700a6a681c0da8ab196769"} Jul 28 05:22:55.773673 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:55.773415 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" event={"ID":"543e0b9a-76d9-4728-8cdb-3ae735ec3dd7","Type":"ContainerStarted","Data":"31e10f4774d1539bba963ca7058d1970b5de9291dc074669202bc054b1f41d79"} Jul 28 05:22:55.793243 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:55.793196 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xxvrw" podStartSLOduration=9.297158122 podStartE2EDuration="38.793182932s" totalImagesPullingTime="29.49602481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:55.791698699 +0000 UTC m=+38.723733863" watchObservedRunningTime="2026-07-28 05:22:55.793182932 +0000 UTC m=+38.725218096" Jul 28 05:22:55.912103 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:55.912072 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:56.012130 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:56.012054 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:56.059641 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:56.059612 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:56.527898 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:56.527871 2529 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:56.606491 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:56.606447 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:22:56.606654 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:56.606623 2529 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:22:56.606700 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:22:56.606689 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls podName:c0855a53-29d0-4bad-95a4-e0696e3357c4 nodeName:}" failed. No retries permitted until 2026-07-28 05:23:00.606671983 +0000 UTC m=+43.538707126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls") pod "dns-default-n7tqp" (UID: "c0855a53-29d0-4bad-95a4-e0696e3357c4") : secret "dns-default-metrics-tls" not found Jul 28 05:22:56.777448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:56.777410 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87bqg" event={"ID":"f4600925-15b9-4d3e-8329-dfa4d59fe6f3","Type":"ContainerStarted","Data":"1930e30bc48de241c82b455a4af3e0d033c28b5fa2b4d9f97f7d3161c4dcab2a"} Jul 28 05:22:56.795572 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:22:56.795470 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-87bqg" podStartSLOduration=2.475308489 podStartE2EDuration="4.79545516s" totalImagesPullingTime="2.320146671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:56.794617934 +0000 UTC m=+39.726653085" watchObservedRunningTime="2026-07-28 05:22:56.79545516 +0000 UTC m=+39.727490319" Jul 28 05:23:00.638292 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:00.638251 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:23:00.638704 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:00.638406 2529 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:00.638704 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:00.638471 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls podName:c0855a53-29d0-4bad-95a4-e0696e3357c4 nodeName:}" failed. No retries permitted until 2026-07-28 05:23:08.638455821 +0000 UTC m=+51.570490960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls") pod "dns-default-n7tqp" (UID: "c0855a53-29d0-4bad-95a4-e0696e3357c4") : secret "dns-default-metrics-tls" not found Jul 28 05:23:08.695749 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:08.695708 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:23:08.696281 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:08.695874 2529 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:08.696281 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:08.695966 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls podName:c0855a53-29d0-4bad-95a4-e0696e3357c4 nodeName:}" failed. No retries permitted until 2026-07-28 05:23:24.695944774 +0000 UTC m=+67.627979929 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls") pod "dns-default-n7tqp" (UID: "c0855a53-29d0-4bad-95a4-e0696e3357c4") : secret "dns-default-metrics-tls" not found Jul 28 05:23:17.753999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:17.753972 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpmnb" Jul 28 05:23:23.192401 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.192348 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:23:23.192401 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.192416 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:23:23.194850 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.194827 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 05:23:23.194905 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.194880 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 05:23:23.202749 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:23.202729 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 05:23:23.202842 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:23.202788 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:24:27.202770566 +0000 UTC m=+130.134805705 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : secret "metrics-daemon-secret" not found Jul 28 05:23:23.206419 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.206397 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ce96221-80e8-4768-a905-818462349deb-original-pull-secret\") pod \"global-pull-secret-syncer-r52w4\" (UID: \"7ce96221-80e8-4768-a905-818462349deb\") " pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:23:23.293662 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.293619 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:23:23.295651 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.295628 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r52w4" Jul 28 05:23:23.295966 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.295949 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 05:23:23.306448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.306427 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 05:23:23.317456 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.317437 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndvx\" (UniqueName: \"kubernetes.io/projected/f678dcbc-bc33-443a-b1bb-e119d0818a10-kube-api-access-qndvx\") pod \"network-check-target-srm2m\" (UID: \"f678dcbc-bc33-443a-b1bb-e119d0818a10\") " pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:23:23.414315 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.414285 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r52w4"] Jul 28 05:23:23.417835 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:23:23.417809 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce96221_80e8_4768_a905_818462349deb.slice/crio-4a217b011ec608a28d4a363a5175cc04af15d74a45a83a1779c3730d560d8966 WatchSource:0}: Error finding container 4a217b011ec608a28d4a363a5175cc04af15d74a45a83a1779c3730d560d8966: Status 404 returned error can't find the container with id 4a217b011ec608a28d4a363a5175cc04af15d74a45a83a1779c3730d560d8966 Jul 28 05:23:23.591829 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.591754 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqsrp\"" Jul 28 05:23:23.600564 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.600540 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:23:23.717011 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.716981 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-srm2m"] Jul 28 05:23:23.720125 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:23:23.720099 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf678dcbc_bc33_443a_b1bb_e119d0818a10.slice/crio-af3992c425726ba69531e0718fd2e0bce5a0a00732a6ed017c639e4b3ac86bfd WatchSource:0}: Error finding container af3992c425726ba69531e0718fd2e0bce5a0a00732a6ed017c639e4b3ac86bfd: Status 404 returned error can't find the container with id af3992c425726ba69531e0718fd2e0bce5a0a00732a6ed017c639e4b3ac86bfd Jul 28 05:23:23.840245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.840206 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r52w4" event={"ID":"7ce96221-80e8-4768-a905-818462349deb","Type":"ContainerStarted","Data":"4a217b011ec608a28d4a363a5175cc04af15d74a45a83a1779c3730d560d8966"} Jul 28 05:23:23.841317 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:23.841290 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-srm2m" event={"ID":"f678dcbc-bc33-443a-b1bb-e119d0818a10","Type":"ContainerStarted","Data":"af3992c425726ba69531e0718fd2e0bce5a0a00732a6ed017c639e4b3ac86bfd"} Jul 28 05:23:24.708400 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:24.708357 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:23:24.708840 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:24.708507 2529 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:24.708840 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:24.708596 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls podName:c0855a53-29d0-4bad-95a4-e0696e3357c4 nodeName:}" failed. No retries permitted until 2026-07-28 05:23:56.708576461 +0000 UTC m=+99.640611602 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls") pod "dns-default-n7tqp" (UID: "c0855a53-29d0-4bad-95a4-e0696e3357c4") : secret "dns-default-metrics-tls" not found Jul 28 05:23:28.855660 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:28.855627 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r52w4" event={"ID":"7ce96221-80e8-4768-a905-818462349deb","Type":"ContainerStarted","Data":"937955d5b2713e8344280ce0b16b1d2180b294b2efc5732026df636f8cd2221c"} Jul 28 05:23:28.857067 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:28.857040 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-srm2m" event={"ID":"f678dcbc-bc33-443a-b1bb-e119d0818a10","Type":"ContainerStarted","Data":"0edf44b6d056dd83af677724f98588de1b5e751109b351b1b733bf3f1075b218"} Jul 28 05:23:28.857184 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:28.857167 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:23:28.869738 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:28.869669 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r52w4" podStartSLOduration=67.159338632 podStartE2EDuration="1m11.869653755s" totalImagesPullingTime="4.710315123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:23:28.869307295 +0000 UTC m=+71.801342456" watchObservedRunningTime="2026-07-28 05:23:28.869653755 +0000 UTC m=+71.801688916" Jul 28 05:23:28.882503 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:28.882443 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-srm2m" podStartSLOduration=67.477854017 podStartE2EDuration="1m11.882428106s" totalImagesPullingTime="4.404574089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:23:28.882315486 +0000 UTC m=+71.814350649" watchObservedRunningTime="2026-07-28 05:23:28.882428106 +0000 UTC m=+71.814463268" Jul 28 05:23:56.715948 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:56.715902 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:23:56.716362 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:56.716057 2529 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:56.716362 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:23:56.716132 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls podName:c0855a53-29d0-4bad-95a4-e0696e3357c4 nodeName:}" failed. No retries permitted until 2026-07-28 05:25:00.716116397 +0000 UTC m=+163.648151535 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls") pod "dns-default-n7tqp" (UID: "c0855a53-29d0-4bad-95a4-e0696e3357c4") : secret "dns-default-metrics-tls" not found Jul 28 05:23:59.862021 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:23:59.861985 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-srm2m" Jul 28 05:24:27.223210 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:24:27.223161 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:24:27.223734 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:24:27.223330 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 05:24:27.223734 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:24:27.223418 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:26:29.223395795 +0000 UTC m=+252.155430947 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : secret "metrics-daemon-secret" not found Jul 28 05:24:55.921016 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:24:55.920965 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n7tqp" podUID="c0855a53-29d0-4bad-95a4-e0696e3357c4" Jul 28 05:24:56.059351 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:24:56.059286 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7tqp" Jul 28 05:24:56.600142 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:24:56.600085 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:25:00.738374 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:25:00.738333 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:25:00.738796 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:25:00.738479 2529 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:25:00.738796 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:25:00.738560 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls podName:c0855a53-29d0-4bad-95a4-e0696e3357c4 nodeName:}" failed. No retries permitted until 2026-07-28 05:27:02.738542593 +0000 UTC m=+285.670577750 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls") pod "dns-default-n7tqp" (UID: "c0855a53-29d0-4bad-95a4-e0696e3357c4") : secret "dns-default-metrics-tls" not found Jul 28 05:25:08.580680 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:25:08.580640 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:26:29.294286 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:29.294250 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:26:29.294775 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:29.294367 2529 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 05:26:29.294775 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:29.294425 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs podName:a585f5cd-60e1-4a50-acdc-2e508fcbbdca nodeName:}" failed. No retries permitted until 2026-07-28 05:28:31.294412195 +0000 UTC m=+374.226447333 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs") pod "network-metrics-daemon-qntsm" (UID: "a585f5cd-60e1-4a50-acdc-2e508fcbbdca") : secret "metrics-daemon-secret" not found Jul 28 05:26:32.143286 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.143250 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8jw4s"] Jul 28 05:26:32.145147 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.145127 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr"] Jul 28 05:26:32.145293 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.145244 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.146824 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.146801 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-799868648d-lbmdp"] Jul 28 05:26:32.146928 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.146911 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.148398 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.148381 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.148484 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.148400 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 28 05:26:32.149753 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.149731 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 28 05:26:32.149858 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.149762 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 28 05:26:32.149909 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.149861 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 28 05:26:32.150200 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.150178 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-49fch\"" Jul 28 05:26:32.150302 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.150269 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 28 05:26:32.150302 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.150277 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 28 05:26:32.150788 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.150766 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 28 05:26:32.150880 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.150789 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 28 05:26:32.151485 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.151467 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gcp8k\"" Jul 28 05:26:32.151596 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.151488 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nr8nq\"" Jul 28 05:26:32.153745 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.152274 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 05:26:32.153745 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.152347 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 05:26:32.156536 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.154559 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 05:26:32.157967 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.157947 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 28 05:26:32.158999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.158985 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 05:26:32.168865 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.168833 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr"] Jul 28 05:26:32.169485 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.169466 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8jw4s"] Jul 28 05:26:32.171457 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.171431 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799868648d-lbmdp"] Jul 28 05:26:32.214042 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214014 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-certificates\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.214159 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214054 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fbe9002f-266b-4831-9c2c-80d889c2eeab-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr\" (UID: \"fbe9002f-266b-4831-9c2c-80d889c2eeab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.214159 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214082 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.214159 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214106 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzzg\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-kube-api-access-ftzzg\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.214159 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214135 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-trusted-ca\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.214297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214157 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-installation-pull-secrets\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.214297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214199 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7f5fa4b6-32bf-4260-a75f-c28d53326d84-serving-cert\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.214297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214232 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-image-registry-private-configuration\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.214297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214249 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-bound-sa-token\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.214297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214276 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9s6\" (UniqueName: \"kubernetes.io/projected/fbe9002f-266b-4831-9c2c-80d889c2eeab-kube-api-access-7l9s6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr\" (UID: \"fbe9002f-266b-4831-9c2c-80d889c2eeab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.214535 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214303 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f5fa4b6-32bf-4260-a75f-c28d53326d84-trusted-ca\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.214535 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214325 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-499bb\" (UniqueName: \"kubernetes.io/projected/7f5fa4b6-32bf-4260-a75f-c28d53326d84-kube-api-access-499bb\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.214535 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214369 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fbe9002f-266b-4831-9c2c-80d889c2eeab-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr\" (UID: \"fbe9002f-266b-4831-9c2c-80d889c2eeab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.214535 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214405 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e72aad-1324-472d-a83d-1e2cd1353e15-ca-trust-extracted\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.214535 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.214429 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f5fa4b6-32bf-4260-a75f-c28d53326d84-config\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.239813 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.239786 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86"] Jul 28 05:26:32.241506 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.241491 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86" Jul 28 05:26:32.244865 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.244847 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zs82z\"" Jul 28 05:26:32.244967 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.244880 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 28 05:26:32.244967 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.244887 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 28 05:26:32.246102 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.246085 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx"] Jul 28 05:26:32.247786 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.247770 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.253928 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.253610 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 28 05:26:32.253928 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.253617 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 28 05:26:32.253928 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.253664 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 28 05:26:32.253928 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.253670 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-228h7\"" Jul 28 05:26:32.254111 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.253928 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 28 05:26:32.258409 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.258392 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86"] Jul 28 05:26:32.262007 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.261985 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx"] Jul 28 05:26:32.315578 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315555 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92772616-27ea-4ff3-a422-a9c09ddbc1c0-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ff9vx\" (UID: \"92772616-27ea-4ff3-a422-a9c09ddbc1c0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.315683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315586 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9s6\" (UniqueName: \"kubernetes.io/projected/fbe9002f-266b-4831-9c2c-80d889c2eeab-kube-api-access-7l9s6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr\" (UID: \"fbe9002f-266b-4831-9c2c-80d889c2eeab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.315683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315606 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f5fa4b6-32bf-4260-a75f-c28d53326d84-trusted-ca\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.315683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315623 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-499bb\" (UniqueName: \"kubernetes.io/projected/7f5fa4b6-32bf-4260-a75f-c28d53326d84-kube-api-access-499bb\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.315683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315638 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmzw\" (UniqueName: \"kubernetes.io/projected/92772616-27ea-4ff3-a422-a9c09ddbc1c0-kube-api-access-dwmzw\") pod \"service-ca-operator-5d6c75f84f-ff9vx\" (UID: \"92772616-27ea-4ff3-a422-a9c09ddbc1c0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.315683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315683 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fbe9002f-266b-4831-9c2c-80d889c2eeab-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr\" (UID: \"fbe9002f-266b-4831-9c2c-80d889c2eeab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.315924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315701 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e72aad-1324-472d-a83d-1e2cd1353e15-ca-trust-extracted\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.315924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315718 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f5fa4b6-32bf-4260-a75f-c28d53326d84-config\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.315924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315745 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-certificates\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.315924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315777 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fbe9002f-266b-4831-9c2c-80d889c2eeab-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr\" (UID: \"fbe9002f-266b-4831-9c2c-80d889c2eeab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.315924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315802 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75d7\" (UniqueName: \"kubernetes.io/projected/bbb7476a-298d-402f-b619-96146cf2be77-kube-api-access-d75d7\") pod \"volume-data-source-validator-549c6d54cf-7kh86\" (UID: \"bbb7476a-298d-402f-b619-96146cf2be77\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86" Jul 28 05:26:32.315924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315835 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.315924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315866 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzzg\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-kube-api-access-ftzzg\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.315924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315898 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-trusted-ca\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.315924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315923 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-installation-pull-secrets\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.316246 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315949 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7f5fa4b6-32bf-4260-a75f-c28d53326d84-serving-cert\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.316246 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.315971 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92772616-27ea-4ff3-a422-a9c09ddbc1c0-config\") pod \"service-ca-operator-5d6c75f84f-ff9vx\" (UID: \"92772616-27ea-4ff3-a422-a9c09ddbc1c0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.316246 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:32.315992 2529 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 05:26:32.316246 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.316005 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-image-registry-private-configuration\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.316246 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:32.316013 2529 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799868648d-lbmdp: secret "image-registry-tls" not found Jul 28 05:26:32.316246 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.316030 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-bound-sa-token\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.316246 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:32.316091 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls podName:37e72aad-1324-472d-a83d-1e2cd1353e15 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:32.816071723 +0000 UTC m=+255.748106862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls") pod "image-registry-799868648d-lbmdp" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15") : secret "image-registry-tls" not found Jul 28 05:26:32.316484 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.316302 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fbe9002f-266b-4831-9c2c-80d889c2eeab-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr\" (UID: \"fbe9002f-266b-4831-9c2c-80d889c2eeab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.316484 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.316447 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f5fa4b6-32bf-4260-a75f-c28d53326d84-config\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.316597 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.316487 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-certificates\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.316699 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.316679 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f5fa4b6-32bf-4260-a75f-c28d53326d84-trusted-ca\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.316851 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.316828 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e72aad-1324-472d-a83d-1e2cd1353e15-ca-trust-extracted\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.317331 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.317292 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-trusted-ca\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.318653 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.318635 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-installation-pull-secrets\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.318736 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.318657 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7f5fa4b6-32bf-4260-a75f-c28d53326d84-serving-cert\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.318736 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.318705 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-image-registry-private-configuration\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.318835 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.318820 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fbe9002f-266b-4831-9c2c-80d889c2eeab-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr\" (UID: \"fbe9002f-266b-4831-9c2c-80d889c2eeab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.323839 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.323812 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9s6\" (UniqueName: \"kubernetes.io/projected/fbe9002f-266b-4831-9c2c-80d889c2eeab-kube-api-access-7l9s6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr\" (UID: \"fbe9002f-266b-4831-9c2c-80d889c2eeab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.323914 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.323810 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-bound-sa-token\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.324330 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.324314 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-499bb\" (UniqueName: \"kubernetes.io/projected/7f5fa4b6-32bf-4260-a75f-c28d53326d84-kube-api-access-499bb\") pod \"console-operator-584dcb6b54-8jw4s\" (UID: \"7f5fa4b6-32bf-4260-a75f-c28d53326d84\") " pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.324469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.324453 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzzg\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-kube-api-access-ftzzg\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.416612 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.416547 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92772616-27ea-4ff3-a422-a9c09ddbc1c0-config\") pod \"service-ca-operator-5d6c75f84f-ff9vx\" (UID: \"92772616-27ea-4ff3-a422-a9c09ddbc1c0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.416612 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.416591 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92772616-27ea-4ff3-a422-a9c09ddbc1c0-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ff9vx\" (UID: \"92772616-27ea-4ff3-a422-a9c09ddbc1c0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.416761 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.416621 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmzw\" (UniqueName: \"kubernetes.io/projected/92772616-27ea-4ff3-a422-a9c09ddbc1c0-kube-api-access-dwmzw\") pod \"service-ca-operator-5d6c75f84f-ff9vx\" (UID: \"92772616-27ea-4ff3-a422-a9c09ddbc1c0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.416761 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.416686 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75d7\" (UniqueName: \"kubernetes.io/projected/bbb7476a-298d-402f-b619-96146cf2be77-kube-api-access-d75d7\") pod \"volume-data-source-validator-549c6d54cf-7kh86\" (UID: \"bbb7476a-298d-402f-b619-96146cf2be77\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86" Jul 28 05:26:32.417064 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.417041 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92772616-27ea-4ff3-a422-a9c09ddbc1c0-config\") pod \"service-ca-operator-5d6c75f84f-ff9vx\" (UID: \"92772616-27ea-4ff3-a422-a9c09ddbc1c0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.418603 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.418585 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92772616-27ea-4ff3-a422-a9c09ddbc1c0-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ff9vx\" (UID: \"92772616-27ea-4ff3-a422-a9c09ddbc1c0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.424185 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.424158 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmzw\" (UniqueName: \"kubernetes.io/projected/92772616-27ea-4ff3-a422-a9c09ddbc1c0-kube-api-access-dwmzw\") pod \"service-ca-operator-5d6c75f84f-ff9vx\" (UID: \"92772616-27ea-4ff3-a422-a9c09ddbc1c0\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.424390 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.424374 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75d7\" (UniqueName: \"kubernetes.io/projected/bbb7476a-298d-402f-b619-96146cf2be77-kube-api-access-d75d7\") pod \"volume-data-source-validator-549c6d54cf-7kh86\" (UID: \"bbb7476a-298d-402f-b619-96146cf2be77\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86" Jul 28 05:26:32.457192 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.457169 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:32.463853 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.463834 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" Jul 28 05:26:32.550859 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.550834 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86" Jul 28 05:26:32.556787 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.556762 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" Jul 28 05:26:32.592179 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.592138 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8jw4s"] Jul 28 05:26:32.596415 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:26:32.596390 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5fa4b6_32bf_4260_a75f_c28d53326d84.slice/crio-93b34a38364116e695a766a6b4f86bf5c22f1d0b2b2e223155b84701910e843b WatchSource:0}: Error finding container 93b34a38364116e695a766a6b4f86bf5c22f1d0b2b2e223155b84701910e843b: Status 404 returned error can't find the container with id 93b34a38364116e695a766a6b4f86bf5c22f1d0b2b2e223155b84701910e843b Jul 28 05:26:32.610423 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.609982 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr"] Jul 28 05:26:32.616137 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:26:32.616077 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe9002f_266b_4831_9c2c_80d889c2eeab.slice/crio-4b32efcefc6061c9f6a281cc4bc7aa064668c77a695e9e509fde06db9859383e WatchSource:0}: Error finding container 4b32efcefc6061c9f6a281cc4bc7aa064668c77a695e9e509fde06db9859383e: Status 404 returned error can't find the container with id 4b32efcefc6061c9f6a281cc4bc7aa064668c77a695e9e509fde06db9859383e Jul 28 05:26:32.682907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.682796 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86"] Jul 28 05:26:32.686282 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:26:32.686256 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb7476a_298d_402f_b619_96146cf2be77.slice/crio-daeea968251eea0c71f94c16bb31488e77678cd2fc2492277991955c22c4200c WatchSource:0}: Error finding container daeea968251eea0c71f94c16bb31488e77678cd2fc2492277991955c22c4200c: Status 404 returned error can't find the container with id daeea968251eea0c71f94c16bb31488e77678cd2fc2492277991955c22c4200c Jul 28 05:26:32.697267 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.697235 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx"] Jul 28 05:26:32.699703 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:26:32.699684 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92772616_27ea_4ff3_a422_a9c09ddbc1c0.slice/crio-6315d6971184172402eae60cd5358b3470f83682a26548913917cd054affd578 WatchSource:0}: Error finding container 6315d6971184172402eae60cd5358b3470f83682a26548913917cd054affd578: Status 404 returned error can't find the container with id 6315d6971184172402eae60cd5358b3470f83682a26548913917cd054affd578 Jul 28 05:26:32.819858 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:32.819831 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:32.820009 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:32.819947 2529 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 05:26:32.820009 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:32.819958 2529 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799868648d-lbmdp: secret "image-registry-tls" not found Jul 28 05:26:32.820009 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:32.820008 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls podName:37e72aad-1324-472d-a83d-1e2cd1353e15 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:33.819990078 +0000 UTC m=+256.752025216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls") pod "image-registry-799868648d-lbmdp" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15") : secret "image-registry-tls" not found Jul 28 05:26:33.279198 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:33.279162 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" event={"ID":"7f5fa4b6-32bf-4260-a75f-c28d53326d84","Type":"ContainerStarted","Data":"93b34a38364116e695a766a6b4f86bf5c22f1d0b2b2e223155b84701910e843b"} Jul 28 05:26:33.281581 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:33.281486 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" event={"ID":"92772616-27ea-4ff3-a422-a9c09ddbc1c0","Type":"ContainerStarted","Data":"6315d6971184172402eae60cd5358b3470f83682a26548913917cd054affd578"} Jul 28 05:26:33.282948 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:33.282871 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" event={"ID":"fbe9002f-266b-4831-9c2c-80d889c2eeab","Type":"ContainerStarted","Data":"4b32efcefc6061c9f6a281cc4bc7aa064668c77a695e9e509fde06db9859383e"} Jul 28 05:26:33.284651 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:33.284613 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86" event={"ID":"bbb7476a-298d-402f-b619-96146cf2be77","Type":"ContainerStarted","Data":"daeea968251eea0c71f94c16bb31488e77678cd2fc2492277991955c22c4200c"} Jul 28 05:26:33.827578 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:33.827533 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:33.827763 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:33.827746 2529 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 05:26:33.827763 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:33.827763 2529 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799868648d-lbmdp: secret "image-registry-tls" not found Jul 28 05:26:33.827881 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:33.827823 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls podName:37e72aad-1324-472d-a83d-1e2cd1353e15 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:35.827805477 +0000 UTC m=+258.759840620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls") pod "image-registry-799868648d-lbmdp" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15") : secret "image-registry-tls" not found Jul 28 05:26:34.289869 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:34.289830 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86" event={"ID":"bbb7476a-298d-402f-b619-96146cf2be77","Type":"ContainerStarted","Data":"b931b26f02a6975f32a648005b7def7fb54de9e46bcc72ce71ac5dd0ca089e82"} Jul 28 05:26:35.844042 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:35.843593 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:35.844042 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:35.843777 2529 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 05:26:35.844042 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:35.843792 2529 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799868648d-lbmdp: secret "image-registry-tls" not found Jul 28 05:26:35.844042 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:35.843846 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls podName:37e72aad-1324-472d-a83d-1e2cd1353e15 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:39.843827813 +0000 UTC m=+262.775862950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls") pod "image-registry-799868648d-lbmdp" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15") : secret "image-registry-tls" not found Jul 28 05:26:36.298741 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.298702 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" event={"ID":"92772616-27ea-4ff3-a422-a9c09ddbc1c0","Type":"ContainerStarted","Data":"9a116a9fd49e3c74a7aa621146a5f456c986d7ddbddbb439f8fbff17203347ff"} Jul 28 05:26:36.300174 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.300140 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" event={"ID":"fbe9002f-266b-4831-9c2c-80d889c2eeab","Type":"ContainerStarted","Data":"198f8395db3013d15f48657c8c0cafe920a822bc57fced5e2e2c7c8a8945a10f"} Jul 28 05:26:36.301531 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.301498 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/0.log" Jul 28 05:26:36.301622 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.301549 2529 generic.go:356] "Generic (PLEG): container finished" podID="7f5fa4b6-32bf-4260-a75f-c28d53326d84" containerID="066302b596560ed667ef297e0f0a6a04fdc31103b9112b80a894ffbcf6b02418" exitCode=255 Jul 28 05:26:36.301622 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.301594 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" event={"ID":"7f5fa4b6-32bf-4260-a75f-c28d53326d84","Type":"ContainerDied","Data":"066302b596560ed667ef297e0f0a6a04fdc31103b9112b80a894ffbcf6b02418"} Jul 28 05:26:36.301770 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.301758 2529 scope.go:117] "RemoveContainer" containerID="066302b596560ed667ef297e0f0a6a04fdc31103b9112b80a894ffbcf6b02418" Jul 28 05:26:36.326683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.326647 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kh86" podStartSLOduration=3.09620716 podStartE2EDuration="4.326634366s" totalImagesPullingTime="1.230427206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:26:34.304964964 +0000 UTC m=+257.237000125" watchObservedRunningTime="2026-07-28 05:26:36.326634366 +0000 UTC m=+259.258669503" Jul 28 05:26:36.326834 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.326816 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" podStartSLOduration=1.405508237 podStartE2EDuration="4.326810709s" totalImagesPullingTime="2.921302472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:26:36.326208272 +0000 UTC m=+259.258243441" watchObservedRunningTime="2026-07-28 05:26:36.326810709 +0000 UTC m=+259.258845867" Jul 28 05:26:36.334357 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.334337 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9tcqp"] Jul 28 05:26:36.338708 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.338686 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.341285 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.341266 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlh56\"" Jul 28 05:26:36.341649 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.341628 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 05:26:36.341730 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.341694 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 05:26:36.341730 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.341640 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 05:26:36.342356 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.342342 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 05:26:36.355664 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.354938 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9tcqp"] Jul 28 05:26:36.356954 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.356887 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" podStartSLOduration=1.359457814 podStartE2EDuration="4.356870587s" totalImagesPullingTime="2.997412773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:26:36.353659168 +0000 UTC m=+259.285694328" watchObservedRunningTime="2026-07-28 05:26:36.356870587 +0000 UTC m=+259.288905748" Jul 28 05:26:36.447557 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.447528 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.447688 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.447576 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.447688 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.447600 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-data-volume\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.447688 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.447618 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqdsl\" (UniqueName: \"kubernetes.io/projected/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-api-access-xqdsl\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.447688 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.447644 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-crio-socket\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.549013 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.548919 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqdsl\" (UniqueName: \"kubernetes.io/projected/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-api-access-xqdsl\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.549013 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.548972 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-crio-socket\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.549013 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.549017 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.549283 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.549044 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.549283 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.549064 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-data-volume\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.549283 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:36.549149 2529 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:36.549283 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:36.549227 2529 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:36.549283 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:36.549274 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:37.049235685 +0000 UTC m=+259.981270843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : configmap "kube-rbac-proxy" not found Jul 28 05:26:36.549548 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:36.549329 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:37.049299639 +0000 UTC m=+259.981334781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:36.549548 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.549341 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-data-volume\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.549548 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.549145 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-crio-socket\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:36.557540 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:36.557503 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqdsl\" (UniqueName: \"kubernetes.io/projected/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-api-access-xqdsl\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:37.052742 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:37.052708 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:37.053125 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:37.052758 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:37.053125 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:37.052839 2529 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:37.053125 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:37.052871 2529 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:37.053125 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:37.052914 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:38.052898206 +0000 UTC m=+260.984933344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : configmap "kube-rbac-proxy" not found Jul 28 05:26:37.053125 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:37.052928 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:38.052922466 +0000 UTC m=+260.984957603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:37.306372 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:37.306302 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/1.log" Jul 28 05:26:37.306714 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:37.306698 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/0.log" Jul 28 05:26:37.306817 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:37.306736 2529 generic.go:356] "Generic (PLEG): container finished" podID="7f5fa4b6-32bf-4260-a75f-c28d53326d84" containerID="ece0ed7ae772b6eb2fc38c2efe49eb293013d53ae2f79b03ac768fcaadee410d" exitCode=255 Jul 28 05:26:37.306887 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:37.306815 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" event={"ID":"7f5fa4b6-32bf-4260-a75f-c28d53326d84","Type":"ContainerDied","Data":"ece0ed7ae772b6eb2fc38c2efe49eb293013d53ae2f79b03ac768fcaadee410d"} Jul 28 05:26:37.306935 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:37.306893 2529 scope.go:117] "RemoveContainer" containerID="066302b596560ed667ef297e0f0a6a04fdc31103b9112b80a894ffbcf6b02418" Jul 28 05:26:37.307136 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:37.307116 2529 scope.go:117] "RemoveContainer" containerID="ece0ed7ae772b6eb2fc38c2efe49eb293013d53ae2f79b03ac768fcaadee410d" Jul 28 05:26:37.307311 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:37.307294 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8jw4s_openshift-console-operator(7f5fa4b6-32bf-4260-a75f-c28d53326d84)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" podUID="7f5fa4b6-32bf-4260-a75f-c28d53326d84" Jul 28 05:26:38.060689 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:38.060650 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:38.061128 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:38.060761 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:38.061128 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:38.060796 2529 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:38.061128 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:38.060847 2529 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:38.061128 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:38.060858 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:40.060843704 +0000 UTC m=+262.992878847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:38.061128 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:38.060902 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:40.060884668 +0000 UTC m=+262.992919805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : configmap "kube-rbac-proxy" not found Jul 28 05:26:38.127904 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:38.127879 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6cnbv_6e82bb4e-d528-49e5-8c0d-02eb57c2e399/dns-node-resolver/0.log" Jul 28 05:26:38.310603 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:38.310573 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/1.log" Jul 28 05:26:38.310902 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:38.310853 2529 scope.go:117] "RemoveContainer" containerID="ece0ed7ae772b6eb2fc38c2efe49eb293013d53ae2f79b03ac768fcaadee410d" Jul 28 05:26:38.311019 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:38.311002 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8jw4s_openshift-console-operator(7f5fa4b6-32bf-4260-a75f-c28d53326d84)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" podUID="7f5fa4b6-32bf-4260-a75f-c28d53326d84" Jul 28 05:26:39.528731 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:39.528703 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6ff9_9c86d6b2-647e-4409-9fb3-ca3861e7b9d4/node-ca/0.log" Jul 28 05:26:39.874681 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:39.874653 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:39.874842 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:39.874779 2529 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 05:26:39.874842 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:39.874790 2529 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799868648d-lbmdp: secret "image-registry-tls" not found Jul 28 05:26:39.874842 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:39.874837 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls podName:37e72aad-1324-472d-a83d-1e2cd1353e15 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:47.874823619 +0000 UTC m=+270.806858767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls") pod "image-registry-799868648d-lbmdp" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15") : secret "image-registry-tls" not found Jul 28 05:26:39.931783 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:39.931720 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87bqg_f4600925-15b9-4d3e-8329-dfa4d59fe6f3/serve-healthcheck-canary/0.log" Jul 28 05:26:40.075638 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:40.075601 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:40.075798 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:40.075650 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:40.075798 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:40.075740 2529 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:40.075798 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:40.075766 2529 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:40.075896 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:40.075803 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:44.075788596 +0000 UTC m=+267.007823734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : configmap "kube-rbac-proxy" not found Jul 28 05:26:40.075896 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:40.075819 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:44.075808863 +0000 UTC m=+267.007844001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:42.458075 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:42.457995 2529 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:42.458075 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:42.458048 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:26:42.458445 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:42.458377 2529 scope.go:117] "RemoveContainer" containerID="ece0ed7ae772b6eb2fc38c2efe49eb293013d53ae2f79b03ac768fcaadee410d" Jul 28 05:26:42.458551 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:42.458533 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8jw4s_openshift-console-operator(7f5fa4b6-32bf-4260-a75f-c28d53326d84)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" podUID="7f5fa4b6-32bf-4260-a75f-c28d53326d84" Jul 28 05:26:44.103693 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:44.103643 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:44.103693 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:44.103708 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:44.104153 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:44.103773 2529 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:44.104153 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:44.103817 2529 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:44.104153 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:44.103836 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:52.103821738 +0000 UTC m=+275.035856876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : configmap "kube-rbac-proxy" not found Jul 28 05:26:44.104153 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:44.103864 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:52.103853209 +0000 UTC m=+275.035888346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:47.935090 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:47.935057 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:47.937333 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:47.937312 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") pod \"image-registry-799868648d-lbmdp\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:48.068072 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:48.068039 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:48.195004 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:48.194957 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799868648d-lbmdp"] Jul 28 05:26:48.196872 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:26:48.196847 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e72aad_1324_472d_a83d_1e2cd1353e15.slice/crio-6663e6e8d87eefac441bc2174ea46b1141bbac7adb7cdd368a465842fcb05cff WatchSource:0}: Error finding container 6663e6e8d87eefac441bc2174ea46b1141bbac7adb7cdd368a465842fcb05cff: Status 404 returned error can't find the container with id 6663e6e8d87eefac441bc2174ea46b1141bbac7adb7cdd368a465842fcb05cff Jul 28 05:26:48.342923 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:48.342885 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799868648d-lbmdp" event={"ID":"37e72aad-1324-472d-a83d-1e2cd1353e15","Type":"ContainerStarted","Data":"2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb"} Jul 28 05:26:48.343091 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:48.342945 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799868648d-lbmdp" event={"ID":"37e72aad-1324-472d-a83d-1e2cd1353e15","Type":"ContainerStarted","Data":"6663e6e8d87eefac441bc2174ea46b1141bbac7adb7cdd368a465842fcb05cff"} Jul 28 05:26:48.343091 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:48.343000 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:26:48.361373 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:48.361302 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-799868648d-lbmdp" podStartSLOduration=16.361279989 podStartE2EDuration="16.361279989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:26:48.359846571 +0000 UTC m=+271.291881730" watchObservedRunningTime="2026-07-28 05:26:48.361279989 +0000 UTC m=+271.293315151" Jul 28 05:26:52.165892 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:52.165853 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:52.165892 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:52.165899 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:52.166385 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:52.166001 2529 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:52.166385 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:52.166052 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls podName:2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4 nodeName:}" failed. No retries permitted until 2026-07-28 05:27:08.166039272 +0000 UTC m=+291.098074409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9tcqp" (UID: "2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:52.166385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:52.166344 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:26:56.580001 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:56.579973 2529 scope.go:117] "RemoveContainer" containerID="ece0ed7ae772b6eb2fc38c2efe49eb293013d53ae2f79b03ac768fcaadee410d" Jul 28 05:26:57.373293 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:57.373267 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:26:57.373686 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:57.373671 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/1.log" Jul 28 05:26:57.373738 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:57.373702 2529 generic.go:356] "Generic (PLEG): container finished" podID="7f5fa4b6-32bf-4260-a75f-c28d53326d84" containerID="7578bc2c559964eaebb1819eb086f9cc4b3a35d67603a95e151e3ded9c3c9949" exitCode=255 Jul 28 05:26:57.373776 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:57.373743 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" event={"ID":"7f5fa4b6-32bf-4260-a75f-c28d53326d84","Type":"ContainerDied","Data":"7578bc2c559964eaebb1819eb086f9cc4b3a35d67603a95e151e3ded9c3c9949"} Jul 28 05:26:57.373808 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:57.373785 2529 scope.go:117] "RemoveContainer" containerID="ece0ed7ae772b6eb2fc38c2efe49eb293013d53ae2f79b03ac768fcaadee410d" Jul 28 05:26:57.374169 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:57.374147 2529 scope.go:117] "RemoveContainer" containerID="7578bc2c559964eaebb1819eb086f9cc4b3a35d67603a95e151e3ded9c3c9949" Jul 28 05:26:57.374328 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:57.374308 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-8jw4s_openshift-console-operator(7f5fa4b6-32bf-4260-a75f-c28d53326d84)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" podUID="7f5fa4b6-32bf-4260-a75f-c28d53326d84" Jul 28 05:26:58.377879 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:58.377854 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:26:59.059759 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:26:59.059708 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n7tqp" podUID="c0855a53-29d0-4bad-95a4-e0696e3357c4" Jul 28 05:26:59.381102 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:26:59.381077 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7tqp" Jul 28 05:27:00.124057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.124016 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799868648d-lbmdp"] Jul 28 05:27:00.150782 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.150754 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85c655d49c-nqhld"] Jul 28 05:27:00.153147 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.153128 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.165144 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.165122 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c655d49c-nqhld"] Jul 28 05:27:00.231448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.228553 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtz65\" (UniqueName: \"kubernetes.io/projected/f14685e5-9535-4ab2-80fc-063bbe29acdb-kube-api-access-wtz65\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.231448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.228634 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f14685e5-9535-4ab2-80fc-063bbe29acdb-registry-tls\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.231448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.228663 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f14685e5-9535-4ab2-80fc-063bbe29acdb-trusted-ca\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.231448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.228705 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f14685e5-9535-4ab2-80fc-063bbe29acdb-installation-pull-secrets\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.231448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.228740 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f14685e5-9535-4ab2-80fc-063bbe29acdb-registry-certificates\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.231448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.228805 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f14685e5-9535-4ab2-80fc-063bbe29acdb-bound-sa-token\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.231448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.228847 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f14685e5-9535-4ab2-80fc-063bbe29acdb-ca-trust-extracted\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.231448 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.228886 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f14685e5-9535-4ab2-80fc-063bbe29acdb-image-registry-private-configuration\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.329880 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.329844 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f14685e5-9535-4ab2-80fc-063bbe29acdb-bound-sa-token\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.329995 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.329889 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f14685e5-9535-4ab2-80fc-063bbe29acdb-ca-trust-extracted\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.329995 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.329913 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f14685e5-9535-4ab2-80fc-063bbe29acdb-image-registry-private-configuration\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.329995 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.329955 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtz65\" (UniqueName: \"kubernetes.io/projected/f14685e5-9535-4ab2-80fc-063bbe29acdb-kube-api-access-wtz65\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.329995 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.329981 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f14685e5-9535-4ab2-80fc-063bbe29acdb-registry-tls\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.330171 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.329997 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f14685e5-9535-4ab2-80fc-063bbe29acdb-trusted-ca\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.330171 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.330020 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f14685e5-9535-4ab2-80fc-063bbe29acdb-installation-pull-secrets\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.330171 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.330040 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f14685e5-9535-4ab2-80fc-063bbe29acdb-registry-certificates\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.330401 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.330376 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f14685e5-9535-4ab2-80fc-063bbe29acdb-ca-trust-extracted\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.330985 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.330965 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f14685e5-9535-4ab2-80fc-063bbe29acdb-registry-certificates\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.331149 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.331131 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f14685e5-9535-4ab2-80fc-063bbe29acdb-trusted-ca\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.332575 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.332524 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f14685e5-9535-4ab2-80fc-063bbe29acdb-image-registry-private-configuration\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.332856 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.332837 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f14685e5-9535-4ab2-80fc-063bbe29acdb-registry-tls\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.332974 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.332955 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f14685e5-9535-4ab2-80fc-063bbe29acdb-installation-pull-secrets\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.337678 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.337654 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f14685e5-9535-4ab2-80fc-063bbe29acdb-bound-sa-token\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.338274 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.338246 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtz65\" (UniqueName: \"kubernetes.io/projected/f14685e5-9535-4ab2-80fc-063bbe29acdb-kube-api-access-wtz65\") pod \"image-registry-85c655d49c-nqhld\" (UID: \"f14685e5-9535-4ab2-80fc-063bbe29acdb\") " pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.461603 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.461533 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:00.583105 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:00.583074 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c655d49c-nqhld"] Jul 28 05:27:00.584907 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:00.584879 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14685e5_9535_4ab2_80fc_063bbe29acdb.slice/crio-593005859ac2dd74e6d2269808803d32c3b18d2606ffcb25eedd1cd1afa32ad9 WatchSource:0}: Error finding container 593005859ac2dd74e6d2269808803d32c3b18d2606ffcb25eedd1cd1afa32ad9: Status 404 returned error can't find the container with id 593005859ac2dd74e6d2269808803d32c3b18d2606ffcb25eedd1cd1afa32ad9 Jul 28 05:27:01.389033 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:01.388997 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c655d49c-nqhld" event={"ID":"f14685e5-9535-4ab2-80fc-063bbe29acdb","Type":"ContainerStarted","Data":"cf37e63f616443a8b4c6aa2fe01587f1cd9acb64696c099a8938274c948073cc"} Jul 28 05:27:01.389185 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:01.389044 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c655d49c-nqhld" event={"ID":"f14685e5-9535-4ab2-80fc-063bbe29acdb","Type":"ContainerStarted","Data":"593005859ac2dd74e6d2269808803d32c3b18d2606ffcb25eedd1cd1afa32ad9"} Jul 28 05:27:01.389185 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:01.389123 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:01.409081 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:01.409031 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85c655d49c-nqhld" podStartSLOduration=1.4090165479999999 podStartE2EDuration="1.409016548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:01.407902297 +0000 UTC m=+284.339937455" watchObservedRunningTime="2026-07-28 05:27:01.409016548 +0000 UTC m=+284.341051709" Jul 28 05:27:02.457843 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:02.457815 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:27:02.458224 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:02.457863 2529 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:27:02.458224 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:02.458138 2529 scope.go:117] "RemoveContainer" containerID="7578bc2c559964eaebb1819eb086f9cc4b3a35d67603a95e151e3ded9c3c9949" Jul 28 05:27:02.458298 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:27:02.458272 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-8jw4s_openshift-console-operator(7f5fa4b6-32bf-4260-a75f-c28d53326d84)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" podUID="7f5fa4b6-32bf-4260-a75f-c28d53326d84" Jul 28 05:27:02.747887 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:02.747796 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:27:02.750093 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:02.750067 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0855a53-29d0-4bad-95a4-e0696e3357c4-metrics-tls\") pod \"dns-default-n7tqp\" (UID: \"c0855a53-29d0-4bad-95a4-e0696e3357c4\") " pod="openshift-dns/dns-default-n7tqp" Jul 28 05:27:02.984576 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:02.984547 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cs9jc\"" Jul 28 05:27:02.992594 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:02.992560 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7tqp" Jul 28 05:27:03.116618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:03.116554 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n7tqp"] Jul 28 05:27:03.119030 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:03.119001 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0855a53_29d0_4bad_95a4_e0696e3357c4.slice/crio-28fcc1742de386175199896dc63331ef2538b8476997fcc5224527d7400a102c WatchSource:0}: Error finding container 28fcc1742de386175199896dc63331ef2538b8476997fcc5224527d7400a102c: Status 404 returned error can't find the container with id 28fcc1742de386175199896dc63331ef2538b8476997fcc5224527d7400a102c Jul 28 05:27:03.396800 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:03.396771 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n7tqp" event={"ID":"c0855a53-29d0-4bad-95a4-e0696e3357c4","Type":"ContainerStarted","Data":"28fcc1742de386175199896dc63331ef2538b8476997fcc5224527d7400a102c"} Jul 28 05:27:05.405308 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:05.405273 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n7tqp" event={"ID":"c0855a53-29d0-4bad-95a4-e0696e3357c4","Type":"ContainerStarted","Data":"f56751140409d8e6d52a37d518c4c6d0c89f0ca11644201b48b7bd060edba8d1"} Jul 28 05:27:05.405764 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:05.405325 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n7tqp" event={"ID":"c0855a53-29d0-4bad-95a4-e0696e3357c4","Type":"ContainerStarted","Data":"36424f303c58a0e7c804acdf7ed5f49ae0de88d07c944e63840688f1d7b1f0ac"} Jul 28 05:27:05.405764 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:05.405406 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n7tqp" Jul 28 05:27:05.423371 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:05.421839 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n7tqp" podStartSLOduration=251.970037747 podStartE2EDuration="4m13.421822214s" totalImagesPullingTime="1.451784467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:05.420084376 +0000 UTC m=+288.352119535" watchObservedRunningTime="2026-07-28 05:27:05.421822214 +0000 UTC m=+288.353857375" Jul 28 05:27:06.433738 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.433712 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw"] Jul 28 05:27:06.436105 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.436089 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" Jul 28 05:27:06.438209 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.438188 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 28 05:27:06.438299 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.438193 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6vs2m\"" Jul 28 05:27:06.443845 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.443823 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw"] Jul 28 05:27:06.577744 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.577692 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac096c81-397b-446c-83b6-823b10249e8b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-jq5lw\" (UID: \"ac096c81-397b-446c-83b6-823b10249e8b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" Jul 28 05:27:06.678399 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.678365 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac096c81-397b-446c-83b6-823b10249e8b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-jq5lw\" (UID: \"ac096c81-397b-446c-83b6-823b10249e8b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" Jul 28 05:27:06.680686 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.680666 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac096c81-397b-446c-83b6-823b10249e8b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-jq5lw\" (UID: \"ac096c81-397b-446c-83b6-823b10249e8b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" Jul 28 05:27:06.744755 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.744674 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" Jul 28 05:27:06.875816 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:06.875719 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw"] Jul 28 05:27:06.878348 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:06.878310 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac096c81_397b_446c_83b6_823b10249e8b.slice/crio-4b56d1f853835a4e56b98fe00f9488851908302b5b8d8931806e44a8d781ea2e WatchSource:0}: Error finding container 4b56d1f853835a4e56b98fe00f9488851908302b5b8d8931806e44a8d781ea2e: Status 404 returned error can't find the container with id 4b56d1f853835a4e56b98fe00f9488851908302b5b8d8931806e44a8d781ea2e Jul 28 05:27:07.413799 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:07.413751 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" event={"ID":"ac096c81-397b-446c-83b6-823b10249e8b","Type":"ContainerStarted","Data":"4b56d1f853835a4e56b98fe00f9488851908302b5b8d8931806e44a8d781ea2e"} Jul 28 05:27:08.191163 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:08.191069 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:27:08.193297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:08.193280 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9tcqp\" (UID: \"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4\") " pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:27:08.418683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:08.418650 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" event={"ID":"ac096c81-397b-446c-83b6-823b10249e8b","Type":"ContainerStarted","Data":"6ce72bc411193ab252b9cabe12e53c3bba356c6ad605b2557717f9931f44e50b"} Jul 28 05:27:08.418883 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:08.418864 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" Jul 28 05:27:08.423459 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:08.423441 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" Jul 28 05:27:08.432848 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:08.432808 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jq5lw" podStartSLOduration=1.428757504 podStartE2EDuration="2.432793895s" totalImagesPullingTime="1.004036391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:08.431980119 +0000 UTC m=+291.364015282" watchObservedRunningTime="2026-07-28 05:27:08.432793895 +0000 UTC m=+291.364829054" Jul 28 05:27:08.459467 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:08.459411 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9tcqp" Jul 28 05:27:08.579736 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:08.579709 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9tcqp"] Jul 28 05:27:08.583128 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:08.583104 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7d2c11_7a94_45a0_b3ef_bc6b2915bcb4.slice/crio-ad96ce96d43d76237272d3233c586b0d93d20c09970b5e6f4effb60aecf6a262 WatchSource:0}: Error finding container ad96ce96d43d76237272d3233c586b0d93d20c09970b5e6f4effb60aecf6a262: Status 404 returned error can't find the container with id ad96ce96d43d76237272d3233c586b0d93d20c09970b5e6f4effb60aecf6a262 Jul 28 05:27:09.424009 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.423972 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9tcqp" event={"ID":"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4","Type":"ContainerStarted","Data":"ad96ce96d43d76237272d3233c586b0d93d20c09970b5e6f4effb60aecf6a262"} Jul 28 05:27:09.500858 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.500658 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-95z4b"] Jul 28 05:27:09.503756 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.503733 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.506041 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.506017 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 05:27:09.506041 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.506030 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tk2mm\"" Jul 28 05:27:09.506905 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.506886 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 05:27:09.507049 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.506928 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 05:27:09.507049 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.506928 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 05:27:09.507049 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.506951 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 05:27:09.514391 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.513962 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-95z4b"] Jul 28 05:27:09.601730 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.601675 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92cb6b30-f216-461a-97da-1766f27923d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.601905 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.601803 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78gtz\" (UniqueName: \"kubernetes.io/projected/92cb6b30-f216-461a-97da-1766f27923d6-kube-api-access-78gtz\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.601905 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.601849 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/92cb6b30-f216-461a-97da-1766f27923d6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.601905 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.601866 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92cb6b30-f216-461a-97da-1766f27923d6-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.702745 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.702666 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92cb6b30-f216-461a-97da-1766f27923d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.702745 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.702720 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78gtz\" (UniqueName: \"kubernetes.io/projected/92cb6b30-f216-461a-97da-1766f27923d6-kube-api-access-78gtz\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.702745 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.702739 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/92cb6b30-f216-461a-97da-1766f27923d6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.703011 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.702756 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92cb6b30-f216-461a-97da-1766f27923d6-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.703444 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.703413 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92cb6b30-f216-461a-97da-1766f27923d6-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.705317 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.705290 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/92cb6b30-f216-461a-97da-1766f27923d6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.705678 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.705654 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92cb6b30-f216-461a-97da-1766f27923d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.710675 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.710655 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78gtz\" (UniqueName: \"kubernetes.io/projected/92cb6b30-f216-461a-97da-1766f27923d6-kube-api-access-78gtz\") pod \"prometheus-operator-84d69f794d-95z4b\" (UID: \"92cb6b30-f216-461a-97da-1766f27923d6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.815384 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.815346 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" Jul 28 05:27:09.948854 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:09.948831 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-95z4b"] Jul 28 05:27:09.951485 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:09.951454 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cb6b30_f216_461a_97da_1766f27923d6.slice/crio-d1bd584d29a51bbffbd69f4e5cbe9ad6be67ea9c375a926eb3fab590a5aa9920 WatchSource:0}: Error finding container d1bd584d29a51bbffbd69f4e5cbe9ad6be67ea9c375a926eb3fab590a5aa9920: Status 404 returned error can't find the container with id d1bd584d29a51bbffbd69f4e5cbe9ad6be67ea9c375a926eb3fab590a5aa9920 Jul 28 05:27:10.130314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:10.130280 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:27:10.429480 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:10.429442 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" event={"ID":"92cb6b30-f216-461a-97da-1766f27923d6","Type":"ContainerStarted","Data":"d1bd584d29a51bbffbd69f4e5cbe9ad6be67ea9c375a926eb3fab590a5aa9920"} Jul 28 05:27:11.581979 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:27:11.581933 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qntsm" podUID="a585f5cd-60e1-4a50-acdc-2e508fcbbdca" Jul 28 05:27:14.447573 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:14.447295 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" event={"ID":"92cb6b30-f216-461a-97da-1766f27923d6","Type":"ContainerStarted","Data":"3b3b3abb6788210ce2c6adddc98e244c29aab21f0919fa0858227777d8c47676"} Jul 28 05:27:14.447573 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:14.447357 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" event={"ID":"92cb6b30-f216-461a-97da-1766f27923d6","Type":"ContainerStarted","Data":"233b124093e4e04f4c7e2ac9c614fcdbd9d7b01bf3962a6b4925bd8aa5616c55"} Jul 28 05:27:14.449293 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:14.449257 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9tcqp" event={"ID":"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4","Type":"ContainerStarted","Data":"5547efccebb3465bea2c8a285eac89247269fb3c322d5f722eec34fd1668e077"} Jul 28 05:27:15.410935 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:15.410899 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n7tqp" Jul 28 05:27:15.426490 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:15.426441 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-95z4b" podStartSLOduration=2.722488412 podStartE2EDuration="6.426420423s" totalImagesPullingTime="3.703932011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:14.463144119 +0000 UTC m=+297.395179278" watchObservedRunningTime="2026-07-28 05:27:15.426420423 +0000 UTC m=+298.358455586" Jul 28 05:27:15.582066 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:15.582027 2529 scope.go:117] "RemoveContainer" containerID="7578bc2c559964eaebb1819eb086f9cc4b3a35d67603a95e151e3ded9c3c9949" Jul 28 05:27:15.582884 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:27:15.582853 2529 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-8jw4s_openshift-console-operator(7f5fa4b6-32bf-4260-a75f-c28d53326d84)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" podUID="7f5fa4b6-32bf-4260-a75f-c28d53326d84" Jul 28 05:27:16.879721 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.879686 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd"] Jul 28 05:27:16.883116 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.883093 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:16.886793 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.886753 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 05:27:16.886902 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.886869 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9tvls\"" Jul 28 05:27:16.887034 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.887021 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 05:27:16.904478 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.904042 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd"] Jul 28 05:27:16.921485 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.921457 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j6dql"] Jul 28 05:27:16.924988 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.924965 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:16.928728 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.928705 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 05:27:16.928836 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.928765 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 05:27:16.928836 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.928817 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 05:27:16.928945 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:16.928817 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rqggq\"" Jul 28 05:27:17.067450 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067421 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt5n9\" (UniqueName: \"kubernetes.io/projected/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-kube-api-access-lt5n9\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.067701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067454 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-root\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.067701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067470 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-textfile\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.067701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067509 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-metrics-client-ca\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.067701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067546 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20f55225-ec6f-4896-9a6a-dc93a51601ff-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.067701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067560 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-tls\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.067701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067575 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-wtmp\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.067701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067598 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.067701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067645 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-sys\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.067701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067702 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-accelerators-collector-config\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.067993 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067739 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/20f55225-ec6f-4896-9a6a-dc93a51601ff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.067993 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067777 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9ddb\" (UniqueName: \"kubernetes.io/projected/20f55225-ec6f-4896-9a6a-dc93a51601ff-kube-api-access-w9ddb\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.067993 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.067812 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20f55225-ec6f-4896-9a6a-dc93a51601ff-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.168468 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168394 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/20f55225-ec6f-4896-9a6a-dc93a51601ff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.168468 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168432 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9ddb\" (UniqueName: \"kubernetes.io/projected/20f55225-ec6f-4896-9a6a-dc93a51601ff-kube-api-access-w9ddb\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.168468 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168451 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20f55225-ec6f-4896-9a6a-dc93a51601ff-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.168742 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168474 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt5n9\" (UniqueName: \"kubernetes.io/projected/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-kube-api-access-lt5n9\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.168742 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168498 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-root\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.168742 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168530 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-textfile\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.168742 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:27:17.168575 2529 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 28 05:27:17.168742 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168609 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-root\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.168742 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:27:17.168643 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f55225-ec6f-4896-9a6a-dc93a51601ff-openshift-state-metrics-tls podName:20f55225-ec6f-4896-9a6a-dc93a51601ff nodeName:}" failed. No retries permitted until 2026-07-28 05:27:17.668622906 +0000 UTC m=+300.600658047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/20f55225-ec6f-4896-9a6a-dc93a51601ff-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-szrmd" (UID: "20f55225-ec6f-4896-9a6a-dc93a51601ff") : secret "openshift-state-metrics-tls" not found Jul 28 05:27:17.168742 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168669 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-metrics-client-ca\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.168742 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168705 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20f55225-ec6f-4896-9a6a-dc93a51601ff-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.168742 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168732 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-tls\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.169162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168755 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-wtmp\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.169162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168787 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.169162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168818 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-sys\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.169162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168896 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-accelerators-collector-config\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.169162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168899 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-textfile\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.169162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.168955 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-sys\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.169162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.169016 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-wtmp\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.169502 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.169187 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-metrics-client-ca\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.169502 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.169419 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20f55225-ec6f-4896-9a6a-dc93a51601ff-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.169502 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.169487 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-accelerators-collector-config\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.171084 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.171054 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20f55225-ec6f-4896-9a6a-dc93a51601ff-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.171301 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.171284 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-tls\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.171451 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.171429 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.177322 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.177295 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9ddb\" (UniqueName: \"kubernetes.io/projected/20f55225-ec6f-4896-9a6a-dc93a51601ff-kube-api-access-w9ddb\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.178438 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.178416 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt5n9\" (UniqueName: \"kubernetes.io/projected/eaf66664-e8b3-47c8-8d0c-b3dac0d9962a-kube-api-access-lt5n9\") pod \"node-exporter-j6dql\" (UID: \"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a\") " pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.235421 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.235393 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j6dql" Jul 28 05:27:17.246978 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:17.246950 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf66664_e8b3_47c8_8d0c_b3dac0d9962a.slice/crio-44c859fff3ed5ab30ebb4b7167878d69c2a7c5ba6d579b50d5855ab440d13b5c WatchSource:0}: Error finding container 44c859fff3ed5ab30ebb4b7167878d69c2a7c5ba6d579b50d5855ab440d13b5c: Status 404 returned error can't find the container with id 44c859fff3ed5ab30ebb4b7167878d69c2a7c5ba6d579b50d5855ab440d13b5c Jul 28 05:27:17.408823 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.408797 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:27:17.411870 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.411845 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:27:17.418614 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.418561 2529 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 05:27:17.460369 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.460340 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9tcqp" event={"ID":"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4","Type":"ContainerStarted","Data":"8dffa7eeb8da8704fded5547bb05452a50d3e9a0fc13a3b55cb79e2d27e4c0ae"} Jul 28 05:27:17.461396 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.461375 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j6dql" event={"ID":"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a","Type":"ContainerStarted","Data":"44c859fff3ed5ab30ebb4b7167878d69c2a7c5ba6d579b50d5855ab440d13b5c"} Jul 28 05:27:17.673572 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.673487 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/20f55225-ec6f-4896-9a6a-dc93a51601ff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.676066 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.676038 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/20f55225-ec6f-4896-9a6a-dc93a51601ff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-szrmd\" (UID: \"20f55225-ec6f-4896-9a6a-dc93a51601ff\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.795089 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.795053 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9tvls\"" Jul 28 05:27:17.802946 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.802918 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" Jul 28 05:27:17.939028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.938307 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 05:27:17.944076 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.944049 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:17.946885 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.946703 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 05:27:17.946885 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.946752 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 05:27:17.946885 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.946769 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 05:27:17.946885 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.946809 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 05:27:17.947836 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.947139 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 05:27:17.947836 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.947255 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 05:27:17.947836 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.947767 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 05:27:17.948014 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.947860 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 05:27:17.948014 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.947958 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 05:27:17.948184 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.948165 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lmpsx\"" Jul 28 05:27:17.963014 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.962991 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 05:27:17.974082 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:17.974058 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd"] Jul 28 05:27:17.976590 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:17.976565 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f55225_ec6f_4896_9a6a_dc93a51601ff.slice/crio-460436dee26aafe37b8d0950fbe1f7322bf07a741488d649e69fc377926bfbca WatchSource:0}: Error finding container 460436dee26aafe37b8d0950fbe1f7322bf07a741488d649e69fc377926bfbca: Status 404 returned error can't find the container with id 460436dee26aafe37b8d0950fbe1f7322bf07a741488d649e69fc377926bfbca Jul 28 05:27:18.077050 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077021 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-config-volume\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077172 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077100 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpq97\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-kube-api-access-vpq97\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077172 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077139 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077169 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-config-out\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077218 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077245 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077273 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077305 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077641 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077369 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077641 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077407 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-web-config\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077641 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077439 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077641 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077501 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.077641 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.077584 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178044 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178003 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-web-config\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178225 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178061 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178225 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178101 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178225 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178144 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178225 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178190 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-config-volume\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178225 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178217 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpq97\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-kube-api-access-vpq97\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178244 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178276 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-config-out\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178301 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178324 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178349 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178380 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.178431 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.178847 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:27:18.178580 2529 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 28 05:27:18.178847 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:27:18.178643 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-main-tls podName:41d40488-a07e-4bfc-92fb-24f95395ea72 nodeName:}" failed. No retries permitted until 2026-07-28 05:27:18.678622497 +0000 UTC m=+301.610657637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72") : secret "alertmanager-main-tls" not found Jul 28 05:27:18.179129 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.179100 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.179371 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.179346 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.180006 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.179980 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.182220 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.182197 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-config-out\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.183141 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.183079 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.183141 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.183084 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.183288 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.183191 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.183801 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.183454 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.183801 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.183765 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-web-config\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.184278 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.184257 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.185079 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.185056 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-config-volume\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.186961 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.186939 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpq97\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-kube-api-access-vpq97\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.468305 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.468273 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j6dql" event={"ID":"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a","Type":"ContainerStarted","Data":"3c6916bb9cedef33f61b6a7f7704527d00d61768bb1ccd096c346109871af222"} Jul 28 05:27:18.470640 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.470574 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" event={"ID":"20f55225-ec6f-4896-9a6a-dc93a51601ff","Type":"ContainerStarted","Data":"184df2480432b114b8fb8cb9d5bb3d0fac00ce118fd249965e8fa14502a6b68c"} Jul 28 05:27:18.470640 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.470629 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" event={"ID":"20f55225-ec6f-4896-9a6a-dc93a51601ff","Type":"ContainerStarted","Data":"3db2b59a8b3fcf47f6179e481e09ee7a506e037199f33b382ea283ff38b8635b"} Jul 28 05:27:18.470763 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.470657 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" event={"ID":"20f55225-ec6f-4896-9a6a-dc93a51601ff","Type":"ContainerStarted","Data":"460436dee26aafe37b8d0950fbe1f7322bf07a741488d649e69fc377926bfbca"} Jul 28 05:27:18.682134 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.682098 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.684876 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.684853 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:18.856973 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:18.856937 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:27:19.014840 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.014803 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 05:27:19.020132 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:19.020090 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d40488_a07e_4bfc_92fb_24f95395ea72.slice/crio-7b838c67b7ce0d4fd0c59dd8c11607a2a2a3772066891e6a8ef2dd0eeaf7bca1 WatchSource:0}: Error finding container 7b838c67b7ce0d4fd0c59dd8c11607a2a2a3772066891e6a8ef2dd0eeaf7bca1: Status 404 returned error can't find the container with id 7b838c67b7ce0d4fd0c59dd8c11607a2a2a3772066891e6a8ef2dd0eeaf7bca1 Jul 28 05:27:19.022986 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.022705 2529 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 05:27:19.476426 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.476389 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerStarted","Data":"7b838c67b7ce0d4fd0c59dd8c11607a2a2a3772066891e6a8ef2dd0eeaf7bca1"} Jul 28 05:27:19.478003 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.477976 2529 generic.go:356] "Generic (PLEG): container finished" podID="eaf66664-e8b3-47c8-8d0c-b3dac0d9962a" containerID="3c6916bb9cedef33f61b6a7f7704527d00d61768bb1ccd096c346109871af222" exitCode=0 Jul 28 05:27:19.478137 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.478009 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j6dql" event={"ID":"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a","Type":"ContainerDied","Data":"3c6916bb9cedef33f61b6a7f7704527d00d61768bb1ccd096c346109871af222"} Jul 28 05:27:19.808325 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.808234 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5bf4cb454b-kvwft"] Jul 28 05:27:19.821508 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.821485 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:19.825538 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.825471 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 05:27:19.826301 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.826038 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 05:27:19.826301 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.826096 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 05:27:19.826301 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.826156 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 05:27:19.826301 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.826096 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fi9cdjvaes3ma\"" Jul 28 05:27:19.826301 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.826214 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mrxrg\"" Jul 28 05:27:19.826677 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.826654 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 05:27:19.828578 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.828554 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bf4cb454b-kvwft"] Jul 28 05:27:19.992984 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.992948 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:19.993182 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.993019 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-grpc-tls\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:19.993182 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.993046 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:19.993182 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.993072 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:19.993182 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.993149 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:19.993420 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.993195 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-tls\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:19.993420 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.993217 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpd4\" (UniqueName: \"kubernetes.io/projected/77bbca72-d81d-494e-a5c0-b1119403e412-kube-api-access-srpd4\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:19.993420 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:19.993236 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77bbca72-d81d-494e-a5c0-b1119403e412-metrics-client-ca\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.094786 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.094750 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.095253 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.094824 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-grpc-tls\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.095253 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.094857 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.095253 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.094894 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.095253 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.095029 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.095253 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.095072 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-tls\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.095253 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.095099 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpd4\" (UniqueName: \"kubernetes.io/projected/77bbca72-d81d-494e-a5c0-b1119403e412-kube-api-access-srpd4\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.095253 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.095127 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77bbca72-d81d-494e-a5c0-b1119403e412-metrics-client-ca\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.095998 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.095944 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77bbca72-d81d-494e-a5c0-b1119403e412-metrics-client-ca\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.097951 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.097910 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.098099 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.098074 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.098148 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.098130 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-grpc-tls\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.098220 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.098199 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-tls\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.098255 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.098236 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.098402 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.098378 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/77bbca72-d81d-494e-a5c0-b1119403e412-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.102686 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.102657 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpd4\" (UniqueName: \"kubernetes.io/projected/77bbca72-d81d-494e-a5c0-b1119403e412-kube-api-access-srpd4\") pod \"thanos-querier-5bf4cb454b-kvwft\" (UID: \"77bbca72-d81d-494e-a5c0-b1119403e412\") " pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:20.136699 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:20.136665 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:21.145892 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:21.145868 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bf4cb454b-kvwft"] Jul 28 05:27:21.148659 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:21.148632 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bbca72_d81d_494e_a5c0_b1119403e412.slice/crio-a8482e8f83c60ed52bcda0f8f7189727bcc28bbb83fc23f9914e200d79d3494f WatchSource:0}: Error finding container a8482e8f83c60ed52bcda0f8f7189727bcc28bbb83fc23f9914e200d79d3494f: Status 404 returned error can't find the container with id a8482e8f83c60ed52bcda0f8f7189727bcc28bbb83fc23f9914e200d79d3494f Jul 28 05:27:21.488733 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:21.488692 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9tcqp" event={"ID":"2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4","Type":"ContainerStarted","Data":"373e3782fd2f0410faeaaf71cac6c73da6c3fb496c34f520f615e93aeb3f494f"} Jul 28 05:27:21.490585 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:21.490559 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j6dql" event={"ID":"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a","Type":"ContainerStarted","Data":"e0bde18e2fb86a3c1716737df841cf332b9d0f298628189a0f920940f9d5d3e4"} Jul 28 05:27:21.490682 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:21.490613 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j6dql" event={"ID":"eaf66664-e8b3-47c8-8d0c-b3dac0d9962a","Type":"ContainerStarted","Data":"bd3b2bd1f18e0f7009e37cb7c7a92e97ef6f8ce03ed9855feb2f465642e508e3"} Jul 28 05:27:21.492453 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:21.492425 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" event={"ID":"20f55225-ec6f-4896-9a6a-dc93a51601ff","Type":"ContainerStarted","Data":"077c8cc2e38cf6c56682663b2689cc3b9974b3f799c8607e1c8743a7bf39e75c"} Jul 28 05:27:21.493503 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:21.493481 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" event={"ID":"77bbca72-d81d-494e-a5c0-b1119403e412","Type":"ContainerStarted","Data":"a8482e8f83c60ed52bcda0f8f7189727bcc28bbb83fc23f9914e200d79d3494f"} Jul 28 05:27:21.505916 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:21.505878 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9tcqp" podStartSLOduration=33.219089179 podStartE2EDuration="45.50586372s" totalImagesPullingTime="12.286774541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:26:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:21.504280287 +0000 UTC m=+304.436315475" watchObservedRunningTime="2026-07-28 05:27:21.50586372 +0000 UTC m=+304.437898878" Jul 28 05:27:21.520331 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:21.520285 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-szrmd" podStartSLOduration=2.929283346 podStartE2EDuration="5.520271732s" totalImagesPullingTime="2.590988386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:21.519165564 +0000 UTC m=+304.451200723" watchObservedRunningTime="2026-07-28 05:27:21.520271732 +0000 UTC m=+304.452306891" Jul 28 05:27:21.570342 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:21.570300 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j6dql" podStartSLOduration=4.37239297 podStartE2EDuration="5.570285389s" totalImagesPullingTime="1.123967348s" totalInitContainerRuntime="73.925071ms" isStatefulPod=true podCreationTimestamp="2026-07-28 05:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:21.568676055 +0000 UTC m=+304.500711216" watchObservedRunningTime="2026-07-28 05:27:21.570285389 +0000 UTC m=+304.502320547" Jul 28 05:27:22.396667 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:22.396638 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85c655d49c-nqhld" Jul 28 05:27:22.499906 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:22.499873 2529 generic.go:356] "Generic (PLEG): container finished" podID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerID="e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de" exitCode=0 Jul 28 05:27:22.500077 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:22.499942 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerDied","Data":"e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de"} Jul 28 05:27:23.033497 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.033457 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 05:27:23.040127 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.040100 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.042827 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.042804 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wl4lp\"" Jul 28 05:27:23.042949 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.042868 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 05:27:23.045389 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.044972 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 05:27:23.045389 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045070 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 05:27:23.045389 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045202 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 05:27:23.045389 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045210 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 05:27:23.045389 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045250 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 05:27:23.045389 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045294 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 05:27:23.045389 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045254 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 05:27:23.045841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045458 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 05:27:23.045841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045575 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 05:27:23.045841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045632 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 05:27:23.045841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.045766 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 05:27:23.046080 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.046061 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4jnim765bmpsh\"" Jul 28 05:27:23.047529 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.047316 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 05:27:23.048307 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.048283 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 05:27:23.054422 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.054039 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 05:27:23.058061 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.058041 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 05:27:23.123302 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123256 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123316 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123426 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-config\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123476 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123464 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123645 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123537 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123645 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123576 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123645 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123594 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123645 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123633 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123801 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123658 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123801 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123701 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123801 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123717 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123801 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123741 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.123801 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123787 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.124008 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123807 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.124008 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123852 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.124008 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123869 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.124008 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123895 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.124008 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123947 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b5bd\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-kube-api-access-9b5bd\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.124008 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123970 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.124008 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.123997 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225114 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225075 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225124 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225158 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225182 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225209 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225245 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225278 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225330 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225368 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225395 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225460 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b5bd\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-kube-api-access-9b5bd\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225486 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225539 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225573 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225606 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225647 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-config\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225689 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225726 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225780 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.225855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.225804 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.226720 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.226659 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.227835 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.227503 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.228243 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.228175 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.228924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.228804 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.229023 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.228946 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.230126 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.229581 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.230126 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.230022 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.230277 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.230198 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.230821 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.230798 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.231721 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.231696 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.234861 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.234823 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.234955 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.234917 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.235016 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.234955 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.235228 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.235210 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.235378 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.235266 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.235842 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.235814 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.236258 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.236239 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-config\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.236336 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.236236 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.237182 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.237148 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.238705 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.238648 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b5bd\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-kube-api-access-9b5bd\") pod \"prometheus-k8s-0\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.356400 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.356360 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:23.506120 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.506090 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" event={"ID":"77bbca72-d81d-494e-a5c0-b1119403e412","Type":"ContainerStarted","Data":"ff398aadf9cfa4c4e9f65935313b7bd6f330fc5e5db59fc306c29465e4b91151"} Jul 28 05:27:23.509756 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:23.509738 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 05:27:23.512239 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:27:23.512212 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9f8235_d134_4696_9fcb_a648bb718282.slice/crio-6f1930a77053ca178812c51cfc63e461cf2d23717dab3d4bf3857cca7d568730 WatchSource:0}: Error finding container 6f1930a77053ca178812c51cfc63e461cf2d23717dab3d4bf3857cca7d568730: Status 404 returned error can't find the container with id 6f1930a77053ca178812c51cfc63e461cf2d23717dab3d4bf3857cca7d568730 Jul 28 05:27:24.512064 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:24.512021 2529 generic.go:356] "Generic (PLEG): container finished" podID="ef9f8235-d134-4696-9fcb-a648bb718282" containerID="088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9" exitCode=0 Jul 28 05:27:24.512558 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:24.512111 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerDied","Data":"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9"} Jul 28 05:27:24.512558 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:24.512213 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerStarted","Data":"6f1930a77053ca178812c51cfc63e461cf2d23717dab3d4bf3857cca7d568730"} Jul 28 05:27:24.514617 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:24.514586 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" event={"ID":"77bbca72-d81d-494e-a5c0-b1119403e412","Type":"ContainerStarted","Data":"cdabdfddfd90d13c939c3725e08fd8a585e3845df3094d02733a8eaa9b974cef"} Jul 28 05:27:24.514737 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:24.514665 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" event={"ID":"77bbca72-d81d-494e-a5c0-b1119403e412","Type":"ContainerStarted","Data":"2795bb533fd95180455b201bdf41c2a93fa84980788b34dd10c8146837cdc759"} Jul 28 05:27:25.142670 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.142623 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-799868648d-lbmdp" podUID="37e72aad-1324-472d-a83d-1e2cd1353e15" containerName="registry" containerID="cri-o://2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb" gracePeriod=30 Jul 28 05:27:25.404732 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.404708 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:27:25.447552 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.447498 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") pod \"37e72aad-1324-472d-a83d-1e2cd1353e15\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " Jul 28 05:27:25.447697 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.447566 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-bound-sa-token\") pod \"37e72aad-1324-472d-a83d-1e2cd1353e15\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " Jul 28 05:27:25.447697 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.447593 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftzzg\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-kube-api-access-ftzzg\") pod \"37e72aad-1324-472d-a83d-1e2cd1353e15\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " Jul 28 05:27:25.447697 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.447633 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-installation-pull-secrets\") pod \"37e72aad-1324-472d-a83d-1e2cd1353e15\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " Jul 28 05:27:25.447853 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.447817 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e72aad-1324-472d-a83d-1e2cd1353e15-ca-trust-extracted\") pod \"37e72aad-1324-472d-a83d-1e2cd1353e15\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " Jul 28 05:27:25.448676 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.447909 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-certificates\") pod \"37e72aad-1324-472d-a83d-1e2cd1353e15\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " Jul 28 05:27:25.448676 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.447952 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-trusted-ca\") pod \"37e72aad-1324-472d-a83d-1e2cd1353e15\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " Jul 28 05:27:25.448676 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.447992 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-image-registry-private-configuration\") pod \"37e72aad-1324-472d-a83d-1e2cd1353e15\" (UID: \"37e72aad-1324-472d-a83d-1e2cd1353e15\") " Jul 28 05:27:25.448676 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.448296 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "37e72aad-1324-472d-a83d-1e2cd1353e15" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:27:25.448913 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.448755 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "37e72aad-1324-472d-a83d-1e2cd1353e15" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:27:25.450637 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.450540 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-kube-api-access-ftzzg" (OuterVolumeSpecName: "kube-api-access-ftzzg") pod "37e72aad-1324-472d-a83d-1e2cd1353e15" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15"). InnerVolumeSpecName "kube-api-access-ftzzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:27:25.450637 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.450547 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "37e72aad-1324-472d-a83d-1e2cd1353e15" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:27:25.450834 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.450802 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "37e72aad-1324-472d-a83d-1e2cd1353e15" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:27:25.450892 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.450840 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "37e72aad-1324-472d-a83d-1e2cd1353e15" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:27:25.451346 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.451314 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "37e72aad-1324-472d-a83d-1e2cd1353e15" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:27:25.458787 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.458761 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e72aad-1324-472d-a83d-1e2cd1353e15-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "37e72aad-1324-472d-a83d-1e2cd1353e15" (UID: "37e72aad-1324-472d-a83d-1e2cd1353e15"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:27:25.524223 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.524187 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerStarted","Data":"89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d"} Jul 28 05:27:25.524679 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.524304 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerStarted","Data":"87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c"} Jul 28 05:27:25.524679 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.524358 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerStarted","Data":"864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c"} Jul 28 05:27:25.524679 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.524398 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerStarted","Data":"d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196"} Jul 28 05:27:25.524679 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.524438 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerStarted","Data":"ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56"} Jul 28 05:27:25.524679 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.524479 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerStarted","Data":"68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834"} Jul 28 05:27:25.525468 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.525440 2529 generic.go:356] "Generic (PLEG): container finished" podID="37e72aad-1324-472d-a83d-1e2cd1353e15" containerID="2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb" exitCode=0 Jul 28 05:27:25.525615 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.525475 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799868648d-lbmdp" Jul 28 05:27:25.525615 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.525544 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799868648d-lbmdp" event={"ID":"37e72aad-1324-472d-a83d-1e2cd1353e15","Type":"ContainerDied","Data":"2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb"} Jul 28 05:27:25.525615 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.525586 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799868648d-lbmdp" event={"ID":"37e72aad-1324-472d-a83d-1e2cd1353e15","Type":"ContainerDied","Data":"6663e6e8d87eefac441bc2174ea46b1141bbac7adb7cdd368a465842fcb05cff"} Jul 28 05:27:25.525615 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.525606 2529 scope.go:117] "RemoveContainer" containerID="2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb" Jul 28 05:27:25.529409 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.529385 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" event={"ID":"77bbca72-d81d-494e-a5c0-b1119403e412","Type":"ContainerStarted","Data":"baafcd0be1547cdc93b4e09eea37d0860ea75e4b7ee901ff8ed1c4c43b9c0fe9"} Jul 28 05:27:25.529538 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.529448 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" event={"ID":"77bbca72-d81d-494e-a5c0-b1119403e412","Type":"ContainerStarted","Data":"8bbcf5d76ac5f853ed8e3ef6950f11a7da4a24675592c45a4df798e24e705164"} Jul 28 05:27:25.529538 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.529490 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" event={"ID":"77bbca72-d81d-494e-a5c0-b1119403e412","Type":"ContainerStarted","Data":"fefff5aa505faa67e7e837b16a380a31967c28eb6023698cf693ed3baaacdf17"} Jul 28 05:27:25.529656 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.529630 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:25.536467 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.536450 2529 scope.go:117] "RemoveContainer" containerID="2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb" Jul 28 05:27:25.536769 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:27:25.536743 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb\": container with ID starting with 2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb not found: ID does not exist" containerID="2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb" Jul 28 05:27:25.536850 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.536773 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb"} err="failed to get container status \"2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb\": rpc error: code = NotFound desc = could not find container \"2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb\": container with ID starting with 2ad24841f3af700c6b5c1db8ad5139526555ebe93621f9c5ec2e43cfe63187fb not found: ID does not exist" Jul 28 05:27:25.548669 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.548628 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.852791011 podStartE2EDuration="8.548613967s" totalImagesPullingTime="4.681558283s" totalInitContainerRuntime="14.264673ms" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:25.547239608 +0000 UTC m=+308.479274766" watchObservedRunningTime="2026-07-28 05:27:25.548613967 +0000 UTC m=+308.480649128" Jul 28 05:27:25.550618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.550281 2529 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-trusted-ca\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:27:25.550618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.550307 2529 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-image-registry-private-configuration\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:27:25.550618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.550333 2529 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:27:25.550618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.550352 2529 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-bound-sa-token\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:27:25.550618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.550365 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftzzg\" (UniqueName: \"kubernetes.io/projected/37e72aad-1324-472d-a83d-1e2cd1353e15-kube-api-access-ftzzg\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:27:25.550618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.550380 2529 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e72aad-1324-472d-a83d-1e2cd1353e15-installation-pull-secrets\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:27:25.550618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.550394 2529 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e72aad-1324-472d-a83d-1e2cd1353e15-ca-trust-extracted\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:27:25.550618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.550406 2529 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e72aad-1324-472d-a83d-1e2cd1353e15-registry-certificates\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:27:25.560970 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.560902 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799868648d-lbmdp"] Jul 28 05:27:25.569470 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.569411 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-799868648d-lbmdp"] Jul 28 05:27:25.586815 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.586729 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e72aad-1324-472d-a83d-1e2cd1353e15" path="/var/lib/kubelet/pods/37e72aad-1324-472d-a83d-1e2cd1353e15/volumes" Jul 28 05:27:25.589560 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:25.589494 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" podStartSLOduration=3.233293332 podStartE2EDuration="6.589480249s" totalImagesPullingTime="3.356186917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:25.588555689 +0000 UTC m=+308.520590849" watchObservedRunningTime="2026-07-28 05:27:25.589480249 +0000 UTC m=+308.521515409" Jul 28 05:27:26.580712 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:26.580683 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:27:27.544652 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:27.544629 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerStarted","Data":"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0"} Jul 28 05:27:27.544795 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:27.544776 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerStarted","Data":"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5"} Jul 28 05:27:27.582428 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:27.582404 2529 scope.go:117] "RemoveContainer" containerID="7578bc2c559964eaebb1819eb086f9cc4b3a35d67603a95e151e3ded9c3c9949" Jul 28 05:27:28.551711 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.551675 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerStarted","Data":"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78"} Jul 28 05:27:28.551892 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.551749 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerStarted","Data":"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984"} Jul 28 05:27:28.551892 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.551788 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerStarted","Data":"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5"} Jul 28 05:27:28.551892 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.551827 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerStarted","Data":"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37"} Jul 28 05:27:28.553442 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.553422 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:27:28.553572 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.553465 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" event={"ID":"7f5fa4b6-32bf-4260-a75f-c28d53326d84","Type":"ContainerStarted","Data":"977ed06be71d5464d4f251c0e2a25c43c70100b8b3a0ad69fbe60ea9bf68e4c2"} Jul 28 05:27:28.553748 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.553730 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:27:28.558431 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.558412 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" Jul 28 05:27:28.582636 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.580839 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.696340835 podStartE2EDuration="5.580821807s" totalImagesPullingTime="2.860465688s" totalInitContainerRuntime="24.015284ms" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:28.578248369 +0000 UTC m=+311.510283529" watchObservedRunningTime="2026-07-28 05:27:28.580821807 +0000 UTC m=+311.512856968" Jul 28 05:27:28.593767 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:28.593728 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-8jw4s" podStartSLOduration=53.573844386 podStartE2EDuration="56.593715847s" totalImagesPullingTime="3.019871461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:28.593038212 +0000 UTC m=+311.525073371" watchObservedRunningTime="2026-07-28 05:27:28.593715847 +0000 UTC m=+311.525751006" Jul 28 05:27:31.540240 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:31.540212 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5bf4cb454b-kvwft" Jul 28 05:27:33.357478 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:33.357444 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:27:46.625382 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:46.625335 2529 generic.go:356] "Generic (PLEG): container finished" podID="92772616-27ea-4ff3-a422-a9c09ddbc1c0" containerID="9a116a9fd49e3c74a7aa621146a5f456c986d7ddbddbb439f8fbff17203347ff" exitCode=0 Jul 28 05:27:46.625779 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:46.625404 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" event={"ID":"92772616-27ea-4ff3-a422-a9c09ddbc1c0","Type":"ContainerDied","Data":"9a116a9fd49e3c74a7aa621146a5f456c986d7ddbddbb439f8fbff17203347ff"} Jul 28 05:27:46.625779 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:46.625719 2529 scope.go:117] "RemoveContainer" containerID="9a116a9fd49e3c74a7aa621146a5f456c986d7ddbddbb439f8fbff17203347ff" Jul 28 05:27:47.630603 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:47.630567 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ff9vx" event={"ID":"92772616-27ea-4ff3-a422-a9c09ddbc1c0","Type":"ContainerStarted","Data":"6de4b9ffbb02786e0b7203f85fb53af4ee49853d38885779a033e2909b0eebfe"} Jul 28 05:27:56.670657 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:56.670622 2529 generic.go:356] "Generic (PLEG): container finished" podID="fbe9002f-266b-4831-9c2c-80d889c2eeab" containerID="198f8395db3013d15f48657c8c0cafe920a822bc57fced5e2e2c7c8a8945a10f" exitCode=0 Jul 28 05:27:56.671182 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:56.670695 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" event={"ID":"fbe9002f-266b-4831-9c2c-80d889c2eeab","Type":"ContainerDied","Data":"198f8395db3013d15f48657c8c0cafe920a822bc57fced5e2e2c7c8a8945a10f"} Jul 28 05:27:56.671182 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:56.671046 2529 scope.go:117] "RemoveContainer" containerID="198f8395db3013d15f48657c8c0cafe920a822bc57fced5e2e2c7c8a8945a10f" Jul 28 05:27:57.676728 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:27:57.676692 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr" event={"ID":"fbe9002f-266b-4831-9c2c-80d889c2eeab","Type":"ContainerStarted","Data":"2acff6c4e1c16703036f92bc38bb626cfaf8193e7838dbe5b6d67d97cfb942de"} Jul 28 05:28:23.357149 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:23.357115 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:23.375768 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:23.375745 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:23.800611 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:23.800540 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:31.316870 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:31.316829 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:28:31.319105 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:31.319078 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f5cd-60e1-4a50-acdc-2e508fcbbdca-metrics-certs\") pod \"network-metrics-daemon-qntsm\" (UID: \"a585f5cd-60e1-4a50-acdc-2e508fcbbdca\") " pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:28:31.384332 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:31.384303 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mr6ww\"" Jul 28 05:28:31.392984 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:31.392949 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qntsm" Jul 28 05:28:31.518395 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:31.518366 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qntsm"] Jul 28 05:28:31.521239 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:28:31.521208 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda585f5cd_60e1_4a50_acdc_2e508fcbbdca.slice/crio-3ee56624e823f12811594aa1217b792c2185e6e60bea5891f552db74cb21debc WatchSource:0}: Error finding container 3ee56624e823f12811594aa1217b792c2185e6e60bea5891f552db74cb21debc: Status 404 returned error can't find the container with id 3ee56624e823f12811594aa1217b792c2185e6e60bea5891f552db74cb21debc Jul 28 05:28:31.818547 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:31.818494 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qntsm" event={"ID":"a585f5cd-60e1-4a50-acdc-2e508fcbbdca","Type":"ContainerStarted","Data":"3ee56624e823f12811594aa1217b792c2185e6e60bea5891f552db74cb21debc"} Jul 28 05:28:32.823691 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:32.823650 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qntsm" event={"ID":"a585f5cd-60e1-4a50-acdc-2e508fcbbdca","Type":"ContainerStarted","Data":"0a9a1805a7d81048c0838e9b2881a90a593e981902ad495abacfbcb858c73593"} Jul 28 05:28:32.824166 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:32.823704 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qntsm" event={"ID":"a585f5cd-60e1-4a50-acdc-2e508fcbbdca","Type":"ContainerStarted","Data":"75f702ab135d275d2597d2f7d87980bce1d0f4ddfb41f4510dfc77f5cab13f3b"} Jul 28 05:28:32.838389 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:32.838336 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qntsm" podStartSLOduration=375.065331818 podStartE2EDuration="6m15.838321772s" totalImagesPullingTime="772.989954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:28:32.837298905 +0000 UTC m=+375.769334075" watchObservedRunningTime="2026-07-28 05:28:32.838321772 +0000 UTC m=+375.770356946" Jul 28 05:28:37.073721 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.073692 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 05:28:37.074075 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.074050 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="alertmanager" containerID="cri-o://68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834" gracePeriod=120 Jul 28 05:28:37.074158 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.074117 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy-metric" containerID="cri-o://87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c" gracePeriod=120 Jul 28 05:28:37.074238 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.074152 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy-web" containerID="cri-o://d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196" gracePeriod=120 Jul 28 05:28:37.074238 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.074156 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy" containerID="cri-o://864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c" gracePeriod=120 Jul 28 05:28:37.074238 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.074178 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="prom-label-proxy" containerID="cri-o://89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d" gracePeriod=120 Jul 28 05:28:37.074238 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.074225 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="config-reloader" containerID="cri-o://ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56" gracePeriod=120 Jul 28 05:28:37.846207 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.846176 2529 generic.go:356] "Generic (PLEG): container finished" podID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerID="89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d" exitCode=0 Jul 28 05:28:37.846207 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.846199 2529 generic.go:356] "Generic (PLEG): container finished" podID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerID="864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c" exitCode=0 Jul 28 05:28:37.846207 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.846207 2529 generic.go:356] "Generic (PLEG): container finished" podID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerID="ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56" exitCode=0 Jul 28 05:28:37.846207 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.846216 2529 generic.go:356] "Generic (PLEG): container finished" podID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerID="68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834" exitCode=0 Jul 28 05:28:37.846466 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.846244 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerDied","Data":"89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d"} Jul 28 05:28:37.846466 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.846313 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerDied","Data":"864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c"} Jul 28 05:28:37.846466 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.846340 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerDied","Data":"ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56"} Jul 28 05:28:37.846466 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:37.846363 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerDied","Data":"68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834"} Jul 28 05:28:38.324005 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.323978 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.478741 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.478666 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-web-config\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.478741 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.478735 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-trusted-ca-bundle\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.478916 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.478789 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpq97\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-kube-api-access-vpq97\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.478916 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.478814 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-web\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.478916 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.478857 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.478916 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.478881 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-tls-assets\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.478916 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.478907 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-cluster-tls-config\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.479162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.478939 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-config-out\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.479162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.478982 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-config-volume\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.479162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.479034 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.479162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.479062 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-metrics-client-ca\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.479162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.479082 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-main-db\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.479162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.479129 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-main-tls\") pod \"41d40488-a07e-4bfc-92fb-24f95395ea72\" (UID: \"41d40488-a07e-4bfc-92fb-24f95395ea72\") " Jul 28 05:28:38.479444 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.479163 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:38.479444 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.479390 2529 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.481424 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.481363 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:28:38.481754 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.481722 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:38.482057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.482030 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:38.482510 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.482483 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-kube-api-access-vpq97" (OuterVolumeSpecName: "kube-api-access-vpq97") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "kube-api-access-vpq97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:28:38.483053 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.483027 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:38.483149 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.483081 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:38.483424 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.483390 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:38.483841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.483809 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:28:38.484338 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.484307 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-config-volume" (OuterVolumeSpecName: "config-volume") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:38.484422 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.484382 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-config-out" (OuterVolumeSpecName: "config-out") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:28:38.488456 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.488426 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:38.493597 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.493578 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-web-config" (OuterVolumeSpecName: "web-config") pod "41d40488-a07e-4bfc-92fb-24f95395ea72" (UID: "41d40488-a07e-4bfc-92fb-24f95395ea72"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:38.580531 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580486 2529 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-main-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580543 2529 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-web-config\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580558 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpq97\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-kube-api-access-vpq97\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580572 2529 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580584 2529 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580593 2529 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d40488-a07e-4bfc-92fb-24f95395ea72-tls-assets\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580601 2529 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-cluster-tls-config\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580609 2529 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-config-out\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580617 2529 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-config-volume\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580624 2529 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d40488-a07e-4bfc-92fb-24f95395ea72-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580632 2529 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d40488-a07e-4bfc-92fb-24f95395ea72-metrics-client-ca\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.580919 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.580640 2529 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41d40488-a07e-4bfc-92fb-24f95395ea72-alertmanager-main-db\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:38.853413 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.853364 2529 generic.go:356] "Generic (PLEG): container finished" podID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerID="87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c" exitCode=0 Jul 28 05:28:38.853413 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.853408 2529 generic.go:356] "Generic (PLEG): container finished" podID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerID="d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196" exitCode=0 Jul 28 05:28:38.853628 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.853440 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerDied","Data":"87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c"} Jul 28 05:28:38.853628 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.853460 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.853628 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.853509 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerDied","Data":"d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196"} Jul 28 05:28:38.853628 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.853560 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d40488-a07e-4bfc-92fb-24f95395ea72","Type":"ContainerDied","Data":"7b838c67b7ce0d4fd0c59dd8c11607a2a2a3772066891e6a8ef2dd0eeaf7bca1"} Jul 28 05:28:38.853628 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.853583 2529 scope.go:117] "RemoveContainer" containerID="89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d" Jul 28 05:28:38.861297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.861262 2529 scope.go:117] "RemoveContainer" containerID="87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c" Jul 28 05:28:38.868610 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.868592 2529 scope.go:117] "RemoveContainer" containerID="864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c" Jul 28 05:28:38.875626 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.875606 2529 scope.go:117] "RemoveContainer" containerID="d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196" Jul 28 05:28:38.876223 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.876205 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 05:28:38.882664 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.882642 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 05:28:38.882963 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.882948 2529 scope.go:117] "RemoveContainer" containerID="ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56" Jul 28 05:28:38.889628 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.889614 2529 scope.go:117] "RemoveContainer" containerID="68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834" Jul 28 05:28:38.896242 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.896227 2529 scope.go:117] "RemoveContainer" containerID="e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de" Jul 28 05:28:38.903042 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.903027 2529 scope.go:117] "RemoveContainer" containerID="89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d" Jul 28 05:28:38.903266 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:38.903248 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d\": container with ID starting with 89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d not found: ID does not exist" containerID="89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d" Jul 28 05:28:38.903315 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.903275 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d"} err="failed to get container status \"89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d\": rpc error: code = NotFound desc = could not find container \"89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d\": container with ID starting with 89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d not found: ID does not exist" Jul 28 05:28:38.903315 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.903293 2529 scope.go:117] "RemoveContainer" containerID="87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c" Jul 28 05:28:38.903502 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:38.903485 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c\": container with ID starting with 87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c not found: ID does not exist" containerID="87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c" Jul 28 05:28:38.903571 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.903508 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c"} err="failed to get container status \"87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c\": rpc error: code = NotFound desc = could not find container \"87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c\": container with ID starting with 87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c not found: ID does not exist" Jul 28 05:28:38.903571 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.903539 2529 scope.go:117] "RemoveContainer" containerID="864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c" Jul 28 05:28:38.903759 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:38.903743 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c\": container with ID starting with 864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c not found: ID does not exist" containerID="864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c" Jul 28 05:28:38.903799 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.903762 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c"} err="failed to get container status \"864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c\": rpc error: code = NotFound desc = could not find container \"864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c\": container with ID starting with 864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c not found: ID does not exist" Jul 28 05:28:38.903799 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.903775 2529 scope.go:117] "RemoveContainer" containerID="d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196" Jul 28 05:28:38.904001 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:38.903987 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196\": container with ID starting with d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196 not found: ID does not exist" containerID="d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196" Jul 28 05:28:38.904035 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.904004 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196"} err="failed to get container status \"d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196\": rpc error: code = NotFound desc = could not find container \"d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196\": container with ID starting with d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196 not found: ID does not exist" Jul 28 05:28:38.904035 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.904015 2529 scope.go:117] "RemoveContainer" containerID="ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56" Jul 28 05:28:38.904235 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:38.904218 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56\": container with ID starting with ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56 not found: ID does not exist" containerID="ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56" Jul 28 05:28:38.904275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.904237 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56"} err="failed to get container status \"ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56\": rpc error: code = NotFound desc = could not find container \"ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56\": container with ID starting with ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56 not found: ID does not exist" Jul 28 05:28:38.904275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.904251 2529 scope.go:117] "RemoveContainer" containerID="68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834" Jul 28 05:28:38.904427 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:38.904413 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834\": container with ID starting with 68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834 not found: ID does not exist" containerID="68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834" Jul 28 05:28:38.904466 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.904430 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834"} err="failed to get container status \"68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834\": rpc error: code = NotFound desc = could not find container \"68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834\": container with ID starting with 68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834 not found: ID does not exist" Jul 28 05:28:38.904466 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.904441 2529 scope.go:117] "RemoveContainer" containerID="e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de" Jul 28 05:28:38.904746 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:38.904728 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de\": container with ID starting with e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de not found: ID does not exist" containerID="e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de" Jul 28 05:28:38.904795 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.904751 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de"} err="failed to get container status \"e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de\": rpc error: code = NotFound desc = could not find container \"e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de\": container with ID starting with e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de not found: ID does not exist" Jul 28 05:28:38.904795 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.904766 2529 scope.go:117] "RemoveContainer" containerID="89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d" Jul 28 05:28:38.905014 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.904994 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d"} err="failed to get container status \"89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d\": rpc error: code = NotFound desc = could not find container \"89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d\": container with ID starting with 89122804cbe1746cb0c8eaf66a301c9913ac7d44d2f1551824948a1c960fac7d not found: ID does not exist" Jul 28 05:28:38.905056 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.905015 2529 scope.go:117] "RemoveContainer" containerID="87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c" Jul 28 05:28:38.906101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.905190 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c"} err="failed to get container status \"87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c\": rpc error: code = NotFound desc = could not find container \"87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c\": container with ID starting with 87cfad9a48687f99ded18ddc10e68f49a26680826056da1fe65c3a7e7784c84c not found: ID does not exist" Jul 28 05:28:38.906101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.905259 2529 scope.go:117] "RemoveContainer" containerID="864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c" Jul 28 05:28:38.906101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.905593 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c"} err="failed to get container status \"864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c\": rpc error: code = NotFound desc = could not find container \"864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c\": container with ID starting with 864cac0058ff4f827bbe8ac22669e3ccc216f70d5dc170f1c6ff9e2ee4cb0c5c not found: ID does not exist" Jul 28 05:28:38.906101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.905625 2529 scope.go:117] "RemoveContainer" containerID="d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196" Jul 28 05:28:38.906101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.905879 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196"} err="failed to get container status \"d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196\": rpc error: code = NotFound desc = could not find container \"d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196\": container with ID starting with d4020ce8243de2b6e086534869df547cd8733a5405f2cadfcfd4a926032d5196 not found: ID does not exist" Jul 28 05:28:38.906101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.905901 2529 scope.go:117] "RemoveContainer" containerID="ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56" Jul 28 05:28:38.906425 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.906172 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56"} err="failed to get container status \"ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56\": rpc error: code = NotFound desc = could not find container \"ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56\": container with ID starting with ff5ef79e03310aefbf925018b7436ffad5aa33d532414594216fe89b98762a56 not found: ID does not exist" Jul 28 05:28:38.906425 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.906196 2529 scope.go:117] "RemoveContainer" containerID="68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834" Jul 28 05:28:38.906592 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.906452 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834"} err="failed to get container status \"68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834\": rpc error: code = NotFound desc = could not find container \"68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834\": container with ID starting with 68d9f124d3dd111ac4cd41f426fb25340937ecd169497e417fdeca154801c834 not found: ID does not exist" Jul 28 05:28:38.906592 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.906478 2529 scope.go:117] "RemoveContainer" containerID="e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de" Jul 28 05:28:38.906748 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.906723 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de"} err="failed to get container status \"e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de\": rpc error: code = NotFound desc = could not find container \"e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de\": container with ID starting with e94dfe730ca44983ae32b9a2a42dc1ac66ee48e6f4af3789ce856a55478311de not found: ID does not exist" Jul 28 05:28:38.907373 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907354 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 05:28:38.907738 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907721 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e72aad-1324-472d-a83d-1e2cd1353e15" containerName="registry" Jul 28 05:28:38.907820 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907740 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e72aad-1324-472d-a83d-1e2cd1353e15" containerName="registry" Jul 28 05:28:38.907820 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907757 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="config-reloader" Jul 28 05:28:38.907820 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907765 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="config-reloader" Jul 28 05:28:38.907820 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907782 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy-metric" Jul 28 05:28:38.907820 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907790 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy-metric" Jul 28 05:28:38.907820 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907805 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="alertmanager" Jul 28 05:28:38.907820 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907813 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="alertmanager" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907827 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907834 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907846 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="init-config-reloader" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907855 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="init-config-reloader" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907866 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy-web" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907874 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy-web" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907890 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="prom-label-proxy" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907900 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="prom-label-proxy" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.907991 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.908003 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e72aad-1324-472d-a83d-1e2cd1353e15" containerName="registry" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.908014 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="config-reloader" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.908023 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy-web" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.908032 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="alertmanager" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.908039 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="kube-rbac-proxy-metric" Jul 28 05:28:38.908152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.908048 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" containerName="prom-label-proxy" Jul 28 05:28:38.913235 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.913217 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.915393 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.915377 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 05:28:38.915582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.915555 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 05:28:38.915687 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.915586 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 05:28:38.915687 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.915615 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 05:28:38.915805 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.915695 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 05:28:38.915805 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.915683 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 05:28:38.915805 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.915754 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lmpsx\"" Jul 28 05:28:38.915968 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.915939 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 05:28:38.915968 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.915942 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 05:28:38.920585 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.920567 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 05:28:38.925837 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.925816 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 05:28:38.984593 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984557 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984593 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984598 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984622 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-web-config\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984651 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984678 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984707 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984723 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjqcx\" (UniqueName: \"kubernetes.io/projected/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-kube-api-access-zjqcx\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984738 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-config-out\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984754 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984773 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.984802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984789 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.985081 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984835 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:38.985081 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:38.984871 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085470 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085432 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085470 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085469 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085720 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085546 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085720 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085578 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085720 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085604 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-web-config\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085720 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085637 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085720 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085660 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085973 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085844 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085973 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085883 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjqcx\" (UniqueName: \"kubernetes.io/projected/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-kube-api-access-zjqcx\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085973 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085917 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-config-out\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085973 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085929 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.085973 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085944 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.086208 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.085984 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.086208 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.086009 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.086757 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.086727 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.088472 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.088442 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.088646 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.088616 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.088796 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.088773 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.088863 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.088836 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.088924 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.088867 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.089110 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.089084 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-config-out\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.089218 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.089203 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-web-config\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.089275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.089215 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.089487 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.089474 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.090493 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.090476 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.092759 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.092743 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjqcx\" (UniqueName: \"kubernetes.io/projected/fd0cb4ca-b424-4224-87f5-6eff56c8ff83-kube-api-access-zjqcx\") pod \"alertmanager-main-0\" (UID: \"fd0cb4ca-b424-4224-87f5-6eff56c8ff83\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.223562 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.223509 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 05:28:39.352856 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.352777 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 05:28:39.355601 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:28:39.355563 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0cb4ca_b424_4224_87f5_6eff56c8ff83.slice/crio-41020502d6c213b4a476db067211a11bbba1bd19cd31d80acb11da1019a40fe7 WatchSource:0}: Error finding container 41020502d6c213b4a476db067211a11bbba1bd19cd31d80acb11da1019a40fe7: Status 404 returned error can't find the container with id 41020502d6c213b4a476db067211a11bbba1bd19cd31d80acb11da1019a40fe7 Jul 28 05:28:39.585605 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.585576 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d40488-a07e-4bfc-92fb-24f95395ea72" path="/var/lib/kubelet/pods/41d40488-a07e-4bfc-92fb-24f95395ea72/volumes" Jul 28 05:28:39.859529 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.859487 2529 generic.go:356] "Generic (PLEG): container finished" podID="fd0cb4ca-b424-4224-87f5-6eff56c8ff83" containerID="543b2d036946209b685ae8bbdf7790303ac4975b7824fd9ce04129c94063d526" exitCode=0 Jul 28 05:28:39.859683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.859549 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd0cb4ca-b424-4224-87f5-6eff56c8ff83","Type":"ContainerDied","Data":"543b2d036946209b685ae8bbdf7790303ac4975b7824fd9ce04129c94063d526"} Jul 28 05:28:39.859683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:39.859663 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd0cb4ca-b424-4224-87f5-6eff56c8ff83","Type":"ContainerStarted","Data":"41020502d6c213b4a476db067211a11bbba1bd19cd31d80acb11da1019a40fe7"} Jul 28 05:28:40.866041 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:40.866002 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd0cb4ca-b424-4224-87f5-6eff56c8ff83","Type":"ContainerStarted","Data":"71f1221e1d465b553f912741f10adf3ae87f67b161ec6ab65110f96e686e08a3"} Jul 28 05:28:40.866472 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:40.866066 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd0cb4ca-b424-4224-87f5-6eff56c8ff83","Type":"ContainerStarted","Data":"e532470430017a14d2c465dbe4f9736c990d3fe5d32ec6aa95e40a63a48f0688"} Jul 28 05:28:40.866472 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:40.866090 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd0cb4ca-b424-4224-87f5-6eff56c8ff83","Type":"ContainerStarted","Data":"98ddb2f8828349be2a7c02b524d68134378ed575cde0c4a78c92976f62fb7f25"} Jul 28 05:28:40.866472 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:40.866115 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd0cb4ca-b424-4224-87f5-6eff56c8ff83","Type":"ContainerStarted","Data":"5b46d7ca5d9a65336d5354528fd5469ffae26a46fa2435f661b342e44f1495e7"} Jul 28 05:28:40.866472 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:40.866136 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd0cb4ca-b424-4224-87f5-6eff56c8ff83","Type":"ContainerStarted","Data":"a157b12e4cae0b1db15c9043bcd8befcb1cc4435145208ec9787f3f751b46ec5"} Jul 28 05:28:40.866472 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:40.866161 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd0cb4ca-b424-4224-87f5-6eff56c8ff83","Type":"ContainerStarted","Data":"2ae4931d055199fe82d06c14cbefe64c297a961cd1d8af0eeaee06477be435ed"} Jul 28 05:28:40.893617 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:40.893565 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.8781390140000003 podStartE2EDuration="2.893549683s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.410669ms" isStatefulPod=false podCreationTimestamp="2026-07-28 05:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:28:40.89157152 +0000 UTC m=+383.823606679" watchObservedRunningTime="2026-07-28 05:28:40.893549683 +0000 UTC m=+383.825584840" Jul 28 05:28:41.362809 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.362774 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 05:28:41.363073 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.363047 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="prometheus" containerID="cri-o://900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" gracePeriod=600 Jul 28 05:28:41.363222 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.363074 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="thanos-sidecar" containerID="cri-o://f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" gracePeriod=600 Jul 28 05:28:41.363222 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.363105 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" gracePeriod=600 Jul 28 05:28:41.363222 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.363120 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy-web" containerID="cri-o://bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" gracePeriod=600 Jul 28 05:28:41.363222 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.363144 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="config-reloader" containerID="cri-o://3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" gracePeriod=600 Jul 28 05:28:41.363222 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.363074 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy" containerID="cri-o://df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" gracePeriod=600 Jul 28 05:28:41.617835 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.617771 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:41.710984 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.710956 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b5bd\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-kube-api-access-9b5bd\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711146 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.710994 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711146 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711019 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-metrics-client-ca\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711146 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711048 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-tls\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711146 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711088 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-grpc-tls\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711146 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711114 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-db\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711153 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-trusted-ca-bundle\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711178 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-metrics-client-certs\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711226 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711253 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-0\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711291 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-serving-certs-ca-bundle\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711324 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-tls-assets\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711369 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-web-config\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711408 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-kubelet-serving-ca-bundle\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711440 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-kube-rbac-proxy\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711441 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:41.711979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711469 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-1\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711496 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-config\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711551 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-thanos-prometheus-http-client-file\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711586 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-config-out\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711621 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-2\") pod \"ef9f8235-d134-4696-9fcb-a648bb718282\" (UID: \"ef9f8235-d134-4696-9fcb-a648bb718282\") " Jul 28 05:28:41.711979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711763 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:41.711979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711921 2529 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.711979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.711939 2529 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-metrics-client-ca\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.712365 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.712094 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:28:41.712878 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.712466 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:41.713090 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.713061 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:41.713356 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.713325 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:41.713539 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.713440 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:41.713539 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.713483 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:41.714595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.714567 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-kube-api-access-9b5bd" (OuterVolumeSpecName: "kube-api-access-9b5bd") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "kube-api-access-9b5bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:28:41.715089 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715060 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:41.715206 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715146 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:41.715319 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715293 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:41.715487 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715458 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:28:41.715616 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715590 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:41.715676 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715637 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:41.715732 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715712 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-config-out" (OuterVolumeSpecName: "config-out") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:28:41.715810 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715791 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:41.715904 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715850 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-config" (OuterVolumeSpecName: "config") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:41.715979 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.715964 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:41.725850 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.725830 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-web-config" (OuterVolumeSpecName: "web-config") pod "ef9f8235-d134-4696-9fcb-a648bb718282" (UID: "ef9f8235-d134-4696-9fcb-a648bb718282"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:41.812362 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812323 2529 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812362 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812358 2529 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-metrics-client-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812362 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812369 2529 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812379 2529 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812388 2529 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-tls-assets\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812396 2529 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-web-config\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812404 2529 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812412 2529 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-kube-rbac-proxy\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812422 2529 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812430 2529 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-config\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812438 2529 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812446 2529 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-config-out\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812453 2529 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812461 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b5bd\" (UniqueName: \"kubernetes.io/projected/ef9f8235-d134-4696-9fcb-a648bb718282-kube-api-access-9b5bd\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812472 2529 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812480 2529 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812488 2529 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef9f8235-d134-4696-9fcb-a648bb718282-secret-grpc-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.812584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.812495 2529 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef9f8235-d134-4696-9fcb-a648bb718282-prometheus-k8s-db\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:28:41.875779 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875698 2529 generic.go:356] "Generic (PLEG): container finished" podID="ef9f8235-d134-4696-9fcb-a648bb718282" containerID="c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" exitCode=0 Jul 28 05:28:41.875779 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875722 2529 generic.go:356] "Generic (PLEG): container finished" podID="ef9f8235-d134-4696-9fcb-a648bb718282" containerID="df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" exitCode=0 Jul 28 05:28:41.875779 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875729 2529 generic.go:356] "Generic (PLEG): container finished" podID="ef9f8235-d134-4696-9fcb-a648bb718282" containerID="bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" exitCode=0 Jul 28 05:28:41.875779 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875735 2529 generic.go:356] "Generic (PLEG): container finished" podID="ef9f8235-d134-4696-9fcb-a648bb718282" containerID="f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" exitCode=0 Jul 28 05:28:41.875779 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875740 2529 generic.go:356] "Generic (PLEG): container finished" podID="ef9f8235-d134-4696-9fcb-a648bb718282" containerID="3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" exitCode=0 Jul 28 05:28:41.875779 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875745 2529 generic.go:356] "Generic (PLEG): container finished" podID="ef9f8235-d134-4696-9fcb-a648bb718282" containerID="900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" exitCode=0 Jul 28 05:28:41.876430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875790 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:41.876430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875787 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerDied","Data":"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78"} Jul 28 05:28:41.876430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875952 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerDied","Data":"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984"} Jul 28 05:28:41.876430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.875982 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerDied","Data":"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5"} Jul 28 05:28:41.876430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.876008 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerDied","Data":"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37"} Jul 28 05:28:41.876430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.876015 2529 scope.go:117] "RemoveContainer" containerID="c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" Jul 28 05:28:41.876430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.876038 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerDied","Data":"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0"} Jul 28 05:28:41.876430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.876075 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerDied","Data":"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5"} Jul 28 05:28:41.876430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.876099 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef9f8235-d134-4696-9fcb-a648bb718282","Type":"ContainerDied","Data":"6f1930a77053ca178812c51cfc63e461cf2d23717dab3d4bf3857cca7d568730"} Jul 28 05:28:41.884689 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.884665 2529 scope.go:117] "RemoveContainer" containerID="df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" Jul 28 05:28:41.891629 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.891614 2529 scope.go:117] "RemoveContainer" containerID="bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" Jul 28 05:28:41.898813 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.898799 2529 scope.go:117] "RemoveContainer" containerID="f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" Jul 28 05:28:41.900585 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.900345 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 05:28:41.906109 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.906091 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 05:28:41.910557 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.910537 2529 scope.go:117] "RemoveContainer" containerID="3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" Jul 28 05:28:41.916891 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.916878 2529 scope.go:117] "RemoveContainer" containerID="900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" Jul 28 05:28:41.923993 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.923974 2529 scope.go:117] "RemoveContainer" containerID="088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9" Jul 28 05:28:41.929706 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.929687 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 05:28:41.929986 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.929974 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="thanos-sidecar" Jul 28 05:28:41.930028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.929987 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="thanos-sidecar" Jul 28 05:28:41.930028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.929994 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy-web" Jul 28 05:28:41.930028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.929999 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy-web" Jul 28 05:28:41.930028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930006 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="config-reloader" Jul 28 05:28:41.930028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930011 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="config-reloader" Jul 28 05:28:41.930028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930019 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="prometheus" Jul 28 05:28:41.930028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930024 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="prometheus" Jul 28 05:28:41.930028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930030 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="init-config-reloader" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930035 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="init-config-reloader" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930045 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930051 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930059 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy-thanos" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930064 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy-thanos" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930112 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="thanos-sidecar" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930122 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="prometheus" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930128 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930135 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="config-reloader" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930141 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy-thanos" Jul 28 05:28:41.930256 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930147 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" containerName="kube-rbac-proxy-web" Jul 28 05:28:41.930671 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930621 2529 scope.go:117] "RemoveContainer" containerID="c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" Jul 28 05:28:41.930908 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:41.930889 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": container with ID starting with c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78 not found: ID does not exist" containerID="c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" Jul 28 05:28:41.930943 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930919 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78"} err="failed to get container status \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": rpc error: code = NotFound desc = could not find container \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": container with ID starting with c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78 not found: ID does not exist" Jul 28 05:28:41.930943 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.930939 2529 scope.go:117] "RemoveContainer" containerID="df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" Jul 28 05:28:41.931178 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:41.931161 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": container with ID starting with df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984 not found: ID does not exist" containerID="df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" Jul 28 05:28:41.931242 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.931187 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984"} err="failed to get container status \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": rpc error: code = NotFound desc = could not find container \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": container with ID starting with df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984 not found: ID does not exist" Jul 28 05:28:41.931242 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.931207 2529 scope.go:117] "RemoveContainer" containerID="bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" Jul 28 05:28:41.931417 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:41.931403 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": container with ID starting with bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5 not found: ID does not exist" containerID="bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" Jul 28 05:28:41.931479 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.931424 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5"} err="failed to get container status \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": rpc error: code = NotFound desc = could not find container \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": container with ID starting with bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5 not found: ID does not exist" Jul 28 05:28:41.931479 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.931441 2529 scope.go:117] "RemoveContainer" containerID="f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" Jul 28 05:28:41.931672 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:41.931652 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": container with ID starting with f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37 not found: ID does not exist" containerID="f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" Jul 28 05:28:41.931753 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.931674 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37"} err="failed to get container status \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": rpc error: code = NotFound desc = could not find container \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": container with ID starting with f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37 not found: ID does not exist" Jul 28 05:28:41.931753 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.931687 2529 scope.go:117] "RemoveContainer" containerID="3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" Jul 28 05:28:41.931886 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:41.931872 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": container with ID starting with 3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0 not found: ID does not exist" containerID="3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" Jul 28 05:28:41.931928 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.931889 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0"} err="failed to get container status \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": rpc error: code = NotFound desc = could not find container \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": container with ID starting with 3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0 not found: ID does not exist" Jul 28 05:28:41.931928 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.931900 2529 scope.go:117] "RemoveContainer" containerID="900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" Jul 28 05:28:41.932100 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:41.932082 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": container with ID starting with 900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5 not found: ID does not exist" containerID="900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" Jul 28 05:28:41.932132 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.932105 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5"} err="failed to get container status \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": rpc error: code = NotFound desc = could not find container \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": container with ID starting with 900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5 not found: ID does not exist" Jul 28 05:28:41.932132 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.932119 2529 scope.go:117] "RemoveContainer" containerID="088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9" Jul 28 05:28:41.932361 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:28:41.932344 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": container with ID starting with 088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9 not found: ID does not exist" containerID="088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9" Jul 28 05:28:41.932396 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.932367 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9"} err="failed to get container status \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": rpc error: code = NotFound desc = could not find container \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": container with ID starting with 088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9 not found: ID does not exist" Jul 28 05:28:41.932396 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.932383 2529 scope.go:117] "RemoveContainer" containerID="c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" Jul 28 05:28:41.932646 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.932629 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78"} err="failed to get container status \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": rpc error: code = NotFound desc = could not find container \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": container with ID starting with c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78 not found: ID does not exist" Jul 28 05:28:41.932698 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.932646 2529 scope.go:117] "RemoveContainer" containerID="df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" Jul 28 05:28:41.932858 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.932839 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984"} err="failed to get container status \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": rpc error: code = NotFound desc = could not find container \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": container with ID starting with df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984 not found: ID does not exist" Jul 28 05:28:41.932925 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.932861 2529 scope.go:117] "RemoveContainer" containerID="bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" Jul 28 05:28:41.933083 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.933063 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5"} err="failed to get container status \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": rpc error: code = NotFound desc = could not find container \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": container with ID starting with bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5 not found: ID does not exist" Jul 28 05:28:41.933140 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.933083 2529 scope.go:117] "RemoveContainer" containerID="f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" Jul 28 05:28:41.933353 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.933332 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37"} err="failed to get container status \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": rpc error: code = NotFound desc = could not find container \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": container with ID starting with f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37 not found: ID does not exist" Jul 28 05:28:41.933353 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.933350 2529 scope.go:117] "RemoveContainer" containerID="3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" Jul 28 05:28:41.933643 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.933619 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0"} err="failed to get container status \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": rpc error: code = NotFound desc = could not find container \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": container with ID starting with 3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0 not found: ID does not exist" Jul 28 05:28:41.933643 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.933643 2529 scope.go:117] "RemoveContainer" containerID="900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" Jul 28 05:28:41.933962 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.933872 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5"} err="failed to get container status \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": rpc error: code = NotFound desc = could not find container \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": container with ID starting with 900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5 not found: ID does not exist" Jul 28 05:28:41.933962 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.933896 2529 scope.go:117] "RemoveContainer" containerID="088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9" Jul 28 05:28:41.934275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.934187 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9"} err="failed to get container status \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": rpc error: code = NotFound desc = could not find container \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": container with ID starting with 088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9 not found: ID does not exist" Jul 28 05:28:41.934275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.934211 2529 scope.go:117] "RemoveContainer" containerID="c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.934557 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78"} err="failed to get container status \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": rpc error: code = NotFound desc = could not find container \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": container with ID starting with c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78 not found: ID does not exist" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.934580 2529 scope.go:117] "RemoveContainer" containerID="df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.934880 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984"} err="failed to get container status \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": rpc error: code = NotFound desc = could not find container \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": container with ID starting with df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984 not found: ID does not exist" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.934900 2529 scope.go:117] "RemoveContainer" containerID="bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.935142 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5"} err="failed to get container status \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": rpc error: code = NotFound desc = could not find container \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": container with ID starting with bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5 not found: ID does not exist" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.935161 2529 scope.go:117] "RemoveContainer" containerID="f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.935393 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37"} err="failed to get container status \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": rpc error: code = NotFound desc = could not find container \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": container with ID starting with f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37 not found: ID does not exist" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.935427 2529 scope.go:117] "RemoveContainer" containerID="3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.935693 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0"} err="failed to get container status \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": rpc error: code = NotFound desc = could not find container \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": container with ID starting with 3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0 not found: ID does not exist" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.935712 2529 scope.go:117] "RemoveContainer" containerID="900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.935950 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5"} err="failed to get container status \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": rpc error: code = NotFound desc = could not find container \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": container with ID starting with 900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5 not found: ID does not exist" Jul 28 05:28:41.935999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.935986 2529 scope.go:117] "RemoveContainer" containerID="088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9" Jul 28 05:28:41.936587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.936231 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9"} err="failed to get container status \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": rpc error: code = NotFound desc = could not find container \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": container with ID starting with 088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9 not found: ID does not exist" Jul 28 05:28:41.936587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.936253 2529 scope.go:117] "RemoveContainer" containerID="c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" Jul 28 05:28:41.936587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.936298 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:41.936587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.936484 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78"} err="failed to get container status \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": rpc error: code = NotFound desc = could not find container \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": container with ID starting with c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78 not found: ID does not exist" Jul 28 05:28:41.936587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.936506 2529 scope.go:117] "RemoveContainer" containerID="df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" Jul 28 05:28:41.937112 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.937087 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984"} err="failed to get container status \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": rpc error: code = NotFound desc = could not find container \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": container with ID starting with df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984 not found: ID does not exist" Jul 28 05:28:41.937112 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.937112 2529 scope.go:117] "RemoveContainer" containerID="bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" Jul 28 05:28:41.937350 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.937327 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5"} err="failed to get container status \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": rpc error: code = NotFound desc = could not find container \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": container with ID starting with bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5 not found: ID does not exist" Jul 28 05:28:41.937350 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.937349 2529 scope.go:117] "RemoveContainer" containerID="f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" Jul 28 05:28:41.937618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.937597 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37"} err="failed to get container status \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": rpc error: code = NotFound desc = could not find container \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": container with ID starting with f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37 not found: ID does not exist" Jul 28 05:28:41.937718 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.937621 2529 scope.go:117] "RemoveContainer" containerID="3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" Jul 28 05:28:41.937839 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.937821 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0"} err="failed to get container status \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": rpc error: code = NotFound desc = could not find container \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": container with ID starting with 3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0 not found: ID does not exist" Jul 28 05:28:41.937906 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.937839 2529 scope.go:117] "RemoveContainer" containerID="900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" Jul 28 05:28:41.938057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938043 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5"} err="failed to get container status \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": rpc error: code = NotFound desc = could not find container \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": container with ID starting with 900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5 not found: ID does not exist" Jul 28 05:28:41.938101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938057 2529 scope.go:117] "RemoveContainer" containerID="088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9" Jul 28 05:28:41.938269 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938235 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9"} err="failed to get container status \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": rpc error: code = NotFound desc = could not find container \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": container with ID starting with 088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9 not found: ID does not exist" Jul 28 05:28:41.938269 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938251 2529 scope.go:117] "RemoveContainer" containerID="c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" Jul 28 05:28:41.938454 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938439 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78"} err="failed to get container status \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": rpc error: code = NotFound desc = could not find container \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": container with ID starting with c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78 not found: ID does not exist" Jul 28 05:28:41.938454 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938452 2529 scope.go:117] "RemoveContainer" containerID="df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" Jul 28 05:28:41.938701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938683 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984"} err="failed to get container status \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": rpc error: code = NotFound desc = could not find container \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": container with ID starting with df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984 not found: ID does not exist" Jul 28 05:28:41.938765 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938703 2529 scope.go:117] "RemoveContainer" containerID="bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" Jul 28 05:28:41.938765 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938747 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 05:28:41.938964 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938939 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5"} err="failed to get container status \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": rpc error: code = NotFound desc = could not find container \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": container with ID starting with bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5 not found: ID does not exist" Jul 28 05:28:41.939003 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.938967 2529 scope.go:117] "RemoveContainer" containerID="f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" Jul 28 05:28:41.939174 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.939154 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37"} err="failed to get container status \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": rpc error: code = NotFound desc = could not find container \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": container with ID starting with f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37 not found: ID does not exist" Jul 28 05:28:41.939174 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.939176 2529 scope.go:117] "RemoveContainer" containerID="3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" Jul 28 05:28:41.939402 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.939381 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0"} err="failed to get container status \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": rpc error: code = NotFound desc = could not find container \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": container with ID starting with 3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0 not found: ID does not exist" Jul 28 05:28:41.939477 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.939405 2529 scope.go:117] "RemoveContainer" containerID="900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" Jul 28 05:28:41.939690 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.939672 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5"} err="failed to get container status \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": rpc error: code = NotFound desc = could not find container \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": container with ID starting with 900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5 not found: ID does not exist" Jul 28 05:28:41.939741 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.939692 2529 scope.go:117] "RemoveContainer" containerID="088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9" Jul 28 05:28:41.939741 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.939712 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wl4lp\"" Jul 28 05:28:41.939918 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.939901 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9"} err="failed to get container status \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": rpc error: code = NotFound desc = could not find container \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": container with ID starting with 088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9 not found: ID does not exist" Jul 28 05:28:41.939990 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.939920 2529 scope.go:117] "RemoveContainer" containerID="c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78" Jul 28 05:28:41.940162 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.940144 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78"} err="failed to get container status \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": rpc error: code = NotFound desc = could not find container \"c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78\": container with ID starting with c6e7926c9c98a0fad3aeca2fbe544eea7bbbada51624f1e147d4ef69a09a2d78 not found: ID does not exist" Jul 28 05:28:41.940225 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.940163 2529 scope.go:117] "RemoveContainer" containerID="df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984" Jul 28 05:28:41.940393 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.940375 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984"} err="failed to get container status \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": rpc error: code = NotFound desc = could not find container \"df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984\": container with ID starting with df406c580c2004c6fc03b67121760588c84f21b55be7b91782e05d628786b984 not found: ID does not exist" Jul 28 05:28:41.940462 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.940395 2529 scope.go:117] "RemoveContainer" containerID="bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5" Jul 28 05:28:41.940603 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.940587 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5"} err="failed to get container status \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": rpc error: code = NotFound desc = could not find container \"bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5\": container with ID starting with bf961c321fb7d0fbfa4da6828ab2e560cf689cf25d6a34044b1e1cbe361b0ab5 not found: ID does not exist" Jul 28 05:28:41.940656 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.940603 2529 scope.go:117] "RemoveContainer" containerID="f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37" Jul 28 05:28:41.940799 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.940783 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37"} err="failed to get container status \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": rpc error: code = NotFound desc = could not find container \"f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37\": container with ID starting with f7290645c99506cc6a5ca0c3676115b6868571fafc9b824a1f45236f4c4f8c37 not found: ID does not exist" Jul 28 05:28:41.940866 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.940802 2529 scope.go:117] "RemoveContainer" containerID="3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0" Jul 28 05:28:41.941066 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.941040 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 05:28:41.941066 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.941051 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0"} err="failed to get container status \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": rpc error: code = NotFound desc = could not find container \"3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0\": container with ID starting with 3638e79857521b02b9baa0965678b2c69e15de13c6b425b4d17f61ea494930d0 not found: ID does not exist" Jul 28 05:28:41.941189 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.941071 2529 scope.go:117] "RemoveContainer" containerID="900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5" Jul 28 05:28:41.941317 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.941299 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5"} err="failed to get container status \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": rpc error: code = NotFound desc = could not find container \"900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5\": container with ID starting with 900b0215aa8161fd19da59c0f851aa33698bb1e907d5e71c6e3f464d3285fea5 not found: ID does not exist" Jul 28 05:28:41.941387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.941318 2529 scope.go:117] "RemoveContainer" containerID="088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9" Jul 28 05:28:41.941579 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.941555 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9"} err="failed to get container status \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": rpc error: code = NotFound desc = could not find container \"088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9\": container with ID starting with 088b8a43eed0654a03a9d78a46f7ad654352240132ae89c6d440241f6dfbf6a9 not found: ID does not exist" Jul 28 05:28:41.941674 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.941654 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 05:28:41.941674 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.941658 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 05:28:41.942573 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.942559 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 05:28:41.943017 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.943001 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 05:28:41.943439 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.943424 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 05:28:41.943666 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.943653 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 05:28:41.943736 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.943680 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4jnim765bmpsh\"" Jul 28 05:28:41.944145 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.944129 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 05:28:41.944404 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.944391 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 05:28:41.944583 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.944564 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 05:28:41.945401 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.945385 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 05:28:41.945486 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.945391 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 05:28:41.964125 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.964083 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 05:28:41.967084 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.967067 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 05:28:41.974293 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:41.974274 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 05:28:42.014587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.014554 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.014587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.014591 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.014808 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.014609 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.014808 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.014663 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.014808 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.014697 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.014808 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.014796 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.014949 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.014823 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxlb\" (UniqueName: \"kubernetes.io/projected/69ccc8ce-73fd-47c0-9788-05d38af066cd-kube-api-access-ctxlb\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.014949 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.014864 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-config\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015009 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.014999 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015079 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015061 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015163 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015144 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015210 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015196 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015253 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015228 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/69ccc8ce-73fd-47c0-9788-05d38af066cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015305 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015251 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015354 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015310 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015406 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015362 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015406 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015383 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015406 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015397 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015538 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015436 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/69ccc8ce-73fd-47c0-9788-05d38af066cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.015538 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.015460 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.116643 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.116603 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.116643 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.116647 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.116876 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.116671 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.116876 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.116699 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.116876 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.116724 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.116876 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.116755 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.116876 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.116776 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxlb\" (UniqueName: \"kubernetes.io/projected/69ccc8ce-73fd-47c0-9788-05d38af066cd-kube-api-access-ctxlb\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117120 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117098 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-config\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117176 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117141 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117225 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117176 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117280 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117224 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117280 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117257 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117417 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117287 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/69ccc8ce-73fd-47c0-9788-05d38af066cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117417 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117312 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117417 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117345 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117417 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117385 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117635 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117431 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117635 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117444 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117635 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117456 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117635 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117482 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/69ccc8ce-73fd-47c0-9788-05d38af066cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117635 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117501 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.117635 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.117508 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.118221 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.118191 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.118535 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.118499 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.119458 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.119435 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.119847 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.119822 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.119847 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.119827 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.119996 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.119830 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.119996 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.119827 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/69ccc8ce-73fd-47c0-9788-05d38af066cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.120798 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.120335 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.120798 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.120657 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.120798 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.120710 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.121018 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.120877 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.121148 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.121126 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69ccc8ce-73fd-47c0-9788-05d38af066cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.122041 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.122022 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/69ccc8ce-73fd-47c0-9788-05d38af066cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.122411 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.122386 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.122411 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.122403 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.122540 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.122486 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.122582 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.122557 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/69ccc8ce-73fd-47c0-9788-05d38af066cd-config\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.124832 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.124813 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxlb\" (UniqueName: \"kubernetes.io/projected/69ccc8ce-73fd-47c0-9788-05d38af066cd-kube-api-access-ctxlb\") pod \"prometheus-k8s-0\" (UID: \"69ccc8ce-73fd-47c0-9788-05d38af066cd\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.246766 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.246738 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:28:42.379693 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.379490 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 05:28:42.382490 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:28:42.382466 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ccc8ce_73fd_47c0_9788_05d38af066cd.slice/crio-69bacf26abfc692131bbbfe29dc56cb55f2aafec96ad72ff0e73e90cfcb15260 WatchSource:0}: Error finding container 69bacf26abfc692131bbbfe29dc56cb55f2aafec96ad72ff0e73e90cfcb15260: Status 404 returned error can't find the container with id 69bacf26abfc692131bbbfe29dc56cb55f2aafec96ad72ff0e73e90cfcb15260 Jul 28 05:28:42.882435 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.882400 2529 generic.go:356] "Generic (PLEG): container finished" podID="69ccc8ce-73fd-47c0-9788-05d38af066cd" containerID="1fec3f1a56ee324ac43000265784f95c4d880cac17451d2291c99d290d2a9bfc" exitCode=0 Jul 28 05:28:42.882802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.882450 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69ccc8ce-73fd-47c0-9788-05d38af066cd","Type":"ContainerDied","Data":"1fec3f1a56ee324ac43000265784f95c4d880cac17451d2291c99d290d2a9bfc"} Jul 28 05:28:42.882802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:42.882498 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69ccc8ce-73fd-47c0-9788-05d38af066cd","Type":"ContainerStarted","Data":"69bacf26abfc692131bbbfe29dc56cb55f2aafec96ad72ff0e73e90cfcb15260"} Jul 28 05:28:43.588851 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:43.586510 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9f8235-d134-4696-9fcb-a648bb718282" path="/var/lib/kubelet/pods/ef9f8235-d134-4696-9fcb-a648bb718282/volumes" Jul 28 05:28:43.889358 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:43.889322 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69ccc8ce-73fd-47c0-9788-05d38af066cd","Type":"ContainerStarted","Data":"1a6fbbda26053f3a9c772bfb8b1409f0a70ef3508d1dfd93bbdc14f034ffd5c7"} Jul 28 05:28:43.889908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:43.889412 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69ccc8ce-73fd-47c0-9788-05d38af066cd","Type":"ContainerStarted","Data":"5d739ef16e7a9761a4030802514c0459c806f2c00529c90caa9f009ed0ad4688"} Jul 28 05:28:43.889908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:43.889456 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69ccc8ce-73fd-47c0-9788-05d38af066cd","Type":"ContainerStarted","Data":"ae29becc5e1f89f18a3bb5b7e98ae728be5179e3e753efb495d0a6a5f3d6b938"} Jul 28 05:28:43.889908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:43.889498 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69ccc8ce-73fd-47c0-9788-05d38af066cd","Type":"ContainerStarted","Data":"c8485632600bc86bd508f23cbfc9ec61465b200d1c49f08544cff51f5cefcbd8"} Jul 28 05:28:43.889908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:43.889569 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69ccc8ce-73fd-47c0-9788-05d38af066cd","Type":"ContainerStarted","Data":"cd4bc23ca3648268365eefad38377a1db7efaa1c349d2a5882b1e0e3c6441f4f"} Jul 28 05:28:43.889908 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:43.889606 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69ccc8ce-73fd-47c0-9788-05d38af066cd","Type":"ContainerStarted","Data":"aa4d2abcd2c531b02be862a1b4c8bfaa6474aa594dcc165b58be83b47ce3c408"} Jul 28 05:28:43.916294 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:43.916242 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.896809081 podStartE2EDuration="2.916227569s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.418488ms" isStatefulPod=false podCreationTimestamp="2026-07-28 05:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:28:43.914879045 +0000 UTC m=+386.846914203" watchObservedRunningTime="2026-07-28 05:28:43.916227569 +0000 UTC m=+386.848262730" Jul 28 05:28:47.247499 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:28:47.247462 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:29:42.247600 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:29:42.247569 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:29:42.262786 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:29:42.262765 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:29:43.152531 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:29:43.152488 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 05:32:17.436967 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:32:17.436939 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:32:17.437949 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:32:17.437929 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:37:17.463501 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:37:17.463431 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:37:17.464071 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:37:17.463747 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:38:10.809568 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:10.809537 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-krwzd"] Jul 28 05:38:10.820680 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:10.820649 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-krwzd"] Jul 28 05:38:10.820885 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:10.820792 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-krwzd" Jul 28 05:38:10.823418 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:10.823396 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 05:38:10.823577 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:10.823416 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-m52zh\"" Jul 28 05:38:10.824068 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:10.824049 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 05:38:10.949919 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:10.949884 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkrz\" (UniqueName: \"kubernetes.io/projected/a68907e5-720d-469a-ae45-1f63aa5eb4bc-kube-api-access-zhkrz\") pod \"redis-69bbd798b6-krwzd\" (UID: \"a68907e5-720d-469a-ae45-1f63aa5eb4bc\") " pod="test-ns-feast/redis-69bbd798b6-krwzd" Jul 28 05:38:11.051376 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:11.051343 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkrz\" (UniqueName: \"kubernetes.io/projected/a68907e5-720d-469a-ae45-1f63aa5eb4bc-kube-api-access-zhkrz\") pod \"redis-69bbd798b6-krwzd\" (UID: \"a68907e5-720d-469a-ae45-1f63aa5eb4bc\") " pod="test-ns-feast/redis-69bbd798b6-krwzd" Jul 28 05:38:11.058948 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:11.058923 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkrz\" (UniqueName: \"kubernetes.io/projected/a68907e5-720d-469a-ae45-1f63aa5eb4bc-kube-api-access-zhkrz\") pod \"redis-69bbd798b6-krwzd\" (UID: \"a68907e5-720d-469a-ae45-1f63aa5eb4bc\") " pod="test-ns-feast/redis-69bbd798b6-krwzd" Jul 28 05:38:11.150976 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:11.150947 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-krwzd" Jul 28 05:38:11.273019 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:11.272995 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-krwzd"] Jul 28 05:38:11.275445 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:38:11.275418 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda68907e5_720d_469a_ae45_1f63aa5eb4bc.slice/crio-7580d8b28d50213fc9dfebac4a2bfd56fd879849f322c89d60bed89cee6fdabf WatchSource:0}: Error finding container 7580d8b28d50213fc9dfebac4a2bfd56fd879849f322c89d60bed89cee6fdabf: Status 404 returned error can't find the container with id 7580d8b28d50213fc9dfebac4a2bfd56fd879849f322c89d60bed89cee6fdabf Jul 28 05:38:11.277624 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:11.277604 2529 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 05:38:12.167307 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:12.167270 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-krwzd" event={"ID":"a68907e5-720d-469a-ae45-1f63aa5eb4bc","Type":"ContainerStarted","Data":"7580d8b28d50213fc9dfebac4a2bfd56fd879849f322c89d60bed89cee6fdabf"} Jul 28 05:38:13.172374 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:13.172327 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-krwzd" event={"ID":"a68907e5-720d-469a-ae45-1f63aa5eb4bc","Type":"ContainerStarted","Data":"fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237"} Jul 28 05:38:13.187018 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:38:13.186667 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-krwzd" podStartSLOduration=1.894770786 podStartE2EDuration="3.186648256s" totalImagesPullingTime="1.29187747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:38:13.185307477 +0000 UTC m=+956.117342638" watchObservedRunningTime="2026-07-28 05:38:13.186648256 +0000 UTC m=+956.118683417" Jul 28 05:39:44.491739 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:44.491700 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-krwzd"] Jul 28 05:39:44.492289 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:44.491906 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-krwzd" podUID="a68907e5-720d-469a-ae45-1f63aa5eb4bc" containerName="redis" containerID="cri-o://fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237" gracePeriod=30 Jul 28 05:39:44.735176 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:44.735147 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-krwzd" Jul 28 05:39:44.842634 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:44.842611 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhkrz\" (UniqueName: \"kubernetes.io/projected/a68907e5-720d-469a-ae45-1f63aa5eb4bc-kube-api-access-zhkrz\") pod \"a68907e5-720d-469a-ae45-1f63aa5eb4bc\" (UID: \"a68907e5-720d-469a-ae45-1f63aa5eb4bc\") " Jul 28 05:39:44.844728 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:44.844703 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68907e5-720d-469a-ae45-1f63aa5eb4bc-kube-api-access-zhkrz" (OuterVolumeSpecName: "kube-api-access-zhkrz") pod "a68907e5-720d-469a-ae45-1f63aa5eb4bc" (UID: "a68907e5-720d-469a-ae45-1f63aa5eb4bc"). InnerVolumeSpecName "kube-api-access-zhkrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:39:44.943598 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:44.943509 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkrz\" (UniqueName: \"kubernetes.io/projected/a68907e5-720d-469a-ae45-1f63aa5eb4bc-kube-api-access-zhkrz\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:39:45.561586 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:45.561552 2529 generic.go:356] "Generic (PLEG): container finished" podID="a68907e5-720d-469a-ae45-1f63aa5eb4bc" containerID="fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237" exitCode=0 Jul 28 05:39:45.562010 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:45.561593 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-krwzd" event={"ID":"a68907e5-720d-469a-ae45-1f63aa5eb4bc","Type":"ContainerDied","Data":"fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237"} Jul 28 05:39:45.562010 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:45.561595 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-krwzd" Jul 28 05:39:45.562010 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:45.561620 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-krwzd" event={"ID":"a68907e5-720d-469a-ae45-1f63aa5eb4bc","Type":"ContainerDied","Data":"7580d8b28d50213fc9dfebac4a2bfd56fd879849f322c89d60bed89cee6fdabf"} Jul 28 05:39:45.562010 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:45.561636 2529 scope.go:117] "RemoveContainer" containerID="fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237" Jul 28 05:39:45.571836 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:45.571596 2529 scope.go:117] "RemoveContainer" containerID="fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237" Jul 28 05:39:45.572155 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:39:45.572124 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237\": container with ID starting with fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237 not found: ID does not exist" containerID="fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237" Jul 28 05:39:45.572259 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:45.572162 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237"} err="failed to get container status \"fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237\": rpc error: code = NotFound desc = could not find container \"fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237\": container with ID starting with fcf63fa7a4da7c5ad6c7ff2cff2bd459836c362764d8b6a43e277ec8aa3b8237 not found: ID does not exist" Jul 28 05:39:45.585506 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:45.585487 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-krwzd"] Jul 28 05:39:45.585982 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:45.585965 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-krwzd"] Jul 28 05:39:47.590034 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:39:47.590002 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68907e5-720d-469a-ae45-1f63aa5eb4bc" path="/var/lib/kubelet/pods/a68907e5-720d-469a-ae45-1f63aa5eb4bc/volumes" Jul 28 05:42:17.489093 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:42:17.489021 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:42:17.491233 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:42:17.491213 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:45:49.391925 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.391887 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76"] Jul 28 05:45:49.392440 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.392370 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68907e5-720d-469a-ae45-1f63aa5eb4bc" containerName="redis" Jul 28 05:45:49.392440 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.392389 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68907e5-720d-469a-ae45-1f63aa5eb4bc" containerName="redis" Jul 28 05:45:49.392608 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.392494 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68907e5-720d-469a-ae45-1f63aa5eb4bc" containerName="redis" Jul 28 05:45:49.395820 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.395798 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.397915 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.397889 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 28 05:45:49.398110 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.398089 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 28 05:45:49.398177 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.398143 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-6249n\"" Jul 28 05:45:49.398402 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.398384 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 05:45:49.398669 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.398652 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 05:45:49.398733 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.398694 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 28 05:45:49.398876 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.398861 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 28 05:45:49.406018 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.405998 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76"] Jul 28 05:45:49.464631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.464605 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.464747 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.464635 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.464747 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.464660 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2b5f\" (UniqueName: \"kubernetes.io/projected/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-kube-api-access-n2b5f\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.464747 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.464709 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.464870 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.464824 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.464870 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.464844 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.565933 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.565894 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.566109 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.565969 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.566109 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.565988 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.566109 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.566013 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.566109 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.566030 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.566109 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.566067 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2b5f\" (UniqueName: \"kubernetes.io/projected/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-kube-api-access-n2b5f\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.566354 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:45:49.566157 2529 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 28 05:45:49.566354 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:45:49.566260 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-online-tls podName:a4ac4d92-6b36-40f6-8d24-8d49365c6a5b nodeName:}" failed. No retries permitted until 2026-07-28 05:45:50.06623658 +0000 UTC m=+1412.998271738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-online-tls") pod "feast-simple-feast-setup-777d6cc998-b2f76" (UID: "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b") : secret "feast-simple-feast-setup-online-tls" not found Jul 28 05:45:49.566549 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.566410 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.568351 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.568332 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.568898 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.568879 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.568947 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.568879 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:49.575800 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:49.575774 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2b5f\" (UniqueName: \"kubernetes.io/projected/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-kube-api-access-n2b5f\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:50.069705 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:50.069648 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:50.072110 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:50.072083 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-b2f76\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:50.327670 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:50.327573 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:45:50.459226 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:50.459155 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76"] Jul 28 05:45:50.461776 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:45:50.461749 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4ac4d92_6b36_40f6_8d24_8d49365c6a5b.slice/crio-d918f6a19debe2689fbae12afd3f2b8f01b4b9b00d6950bc3d2dc715d9614de3 WatchSource:0}: Error finding container d918f6a19debe2689fbae12afd3f2b8f01b4b9b00d6950bc3d2dc715d9614de3: Status 404 returned error can't find the container with id d918f6a19debe2689fbae12afd3f2b8f01b4b9b00d6950bc3d2dc715d9614de3 Jul 28 05:45:50.463604 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:50.463587 2529 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 05:45:50.996280 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:45:50.996244 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerStarted","Data":"d918f6a19debe2689fbae12afd3f2b8f01b4b9b00d6950bc3d2dc715d9614de3"} Jul 28 05:46:05.068888 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:05.068843 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerStarted","Data":"00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9"} Jul 28 05:46:09.087245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:09.087162 2529 generic.go:356] "Generic (PLEG): container finished" podID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerID="00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9" exitCode=0 Jul 28 05:46:09.087245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:09.087230 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerDied","Data":"00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9"} Jul 28 05:46:10.094658 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:10.094572 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerStarted","Data":"9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb"} Jul 28 05:46:15.117765 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:15.117732 2529 generic.go:356] "Generic (PLEG): container finished" podID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerID="9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb" exitCode=0 Jul 28 05:46:15.118147 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:15.117814 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerDied","Data":"9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb"} Jul 28 05:46:16.127846 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:16.127806 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerStarted","Data":"4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5"} Jul 28 05:46:16.128323 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:16.127873 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerStarted","Data":"6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad"} Jul 28 05:46:16.128323 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:16.127897 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerStarted","Data":"4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680"} Jul 28 05:46:16.128323 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:16.127918 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerStarted","Data":"e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d"} Jul 28 05:46:16.157088 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:16.157019 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podStartSLOduration=13.484062828999999 podStartE2EDuration="27.156996475s" totalImagesPullingTime="13.672933646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:46:16.155479462 +0000 UTC m=+1439.087514623" watchObservedRunningTime="2026-07-28 05:46:16.156996475 +0000 UTC m=+1439.089031636" Jul 28 05:46:17.327890 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:17.327848 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:17.328358 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:17.327922 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:17.328358 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:17.327943 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:17.328358 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:17.327962 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:17.330005 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:17.329774 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Jul 28 05:46:17.330005 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:17.329834 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 05:46:17.330005 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:17.329939 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused" Jul 28 05:46:17.330235 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:17.330154 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 05:46:20.327949 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:20.327901 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:20.328449 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:20.328293 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Jul 28 05:46:20.328785 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:20.328758 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 05:46:20.328890 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:20.328871 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:20.328956 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:20.328896 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 05:46:20.328956 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:20.328914 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:20.329066 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:20.328957 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:20.329066 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:20.328953 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused" Jul 28 05:46:23.328653 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:23.328495 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:23.329147 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:23.328999 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:23.329147 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:23.329108 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:23.329326 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:23.329307 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:23.333694 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:23.333674 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:24.172129 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:24.172100 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:24.172329 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:24.172249 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:46:24.175086 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:46:24.175067 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:47:17.519430 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:17.519401 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:47:17.522946 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:17.522920 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:47:26.819712 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:26.819675 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76"] Jul 28 05:47:26.820186 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:26.819980 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="registry" containerID="cri-o://e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d" gracePeriod=30 Jul 28 05:47:26.820186 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:26.820079 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="offline" containerID="cri-o://6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad" gracePeriod=30 Jul 28 05:47:26.820186 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:26.820088 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="ui" containerID="cri-o://4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5" gracePeriod=30 Jul 28 05:47:26.820186 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:26.820088 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="online" containerID="cri-o://4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680" gracePeriod=30 Jul 28 05:47:27.114869 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.114833 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt"] Jul 28 05:47:27.117828 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.117803 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.120625 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.120468 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-v4ffh\"" Jul 28 05:47:27.122777 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.122754 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt"] Jul 28 05:47:27.251994 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.251961 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.252159 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.252015 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.252217 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.252160 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.252262 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.252229 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.252302 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.252283 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7v2n\" (UniqueName: \"kubernetes.io/projected/103bd762-0199-4273-a134-6b1528e1d975-kube-api-access-f7v2n\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.252377 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.252361 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/103bd762-0199-4273-a134-6b1528e1d975-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.353834 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.353795 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.353834 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.353835 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7v2n\" (UniqueName: \"kubernetes.io/projected/103bd762-0199-4273-a134-6b1528e1d975-kube-api-access-f7v2n\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.354081 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.353867 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/103bd762-0199-4273-a134-6b1528e1d975-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.354081 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.353890 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.354081 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.353910 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.354081 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.353946 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.354081 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:47:27.354048 2529 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 28 05:47:27.354324 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:47:27.354133 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-online-tls podName:103bd762-0199-4273-a134-6b1528e1d975 nodeName:}" failed. No retries permitted until 2026-07-28 05:47:27.854113371 +0000 UTC m=+1510.786148517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-online-tls") pod "feast-simple-feast-setup-777d6cc998-rl8dt" (UID: "103bd762-0199-4273-a134-6b1528e1d975") : secret "feast-simple-feast-setup-online-tls" not found Jul 28 05:47:27.354390 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.354348 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/103bd762-0199-4273-a134-6b1528e1d975-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.356387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.356359 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.356566 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.356497 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.356645 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.356622 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.362099 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.362074 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7v2n\" (UniqueName: \"kubernetes.io/projected/103bd762-0199-4273-a134-6b1528e1d975-kube-api-access-f7v2n\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.434166 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.434072 2529 generic.go:356] "Generic (PLEG): container finished" podID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerID="4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5" exitCode=0 Jul 28 05:47:27.434166 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.434150 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerDied","Data":"4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5"} Jul 28 05:47:27.858330 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.858304 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:27.860657 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:27.860626 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-rl8dt\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:28.028177 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:28.028139 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:28.166037 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:28.166014 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt"] Jul 28 05:47:28.167362 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:47:28.167332 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod103bd762_0199_4273_a134_6b1528e1d975.slice/crio-e4cf9e37c0185b64dba58d55408eada9ccbebf5b194c3ba136c58b5db660008e WatchSource:0}: Error finding container e4cf9e37c0185b64dba58d55408eada9ccbebf5b194c3ba136c58b5db660008e: Status 404 returned error can't find the container with id e4cf9e37c0185b64dba58d55408eada9ccbebf5b194c3ba136c58b5db660008e Jul 28 05:47:28.442052 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:28.441960 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerStarted","Data":"0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226"} Jul 28 05:47:28.442052 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:28.442015 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerStarted","Data":"e4cf9e37c0185b64dba58d55408eada9ccbebf5b194c3ba136c58b5db660008e"} Jul 28 05:47:28.446323 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:28.446290 2529 generic.go:356] "Generic (PLEG): container finished" podID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerID="4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680" exitCode=0 Jul 28 05:47:28.446495 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:28.446390 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerDied","Data":"4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680"} Jul 28 05:47:33.330057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:33.329971 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 05:47:33.470061 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:33.470027 2529 generic.go:356] "Generic (PLEG): container finished" podID="103bd762-0199-4273-a134-6b1528e1d975" containerID="0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226" exitCode=0 Jul 28 05:47:33.470214 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:33.470104 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerDied","Data":"0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226"} Jul 28 05:47:34.171996 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:34.171954 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 05:47:34.475643 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:34.475551 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerStarted","Data":"c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3"} Jul 28 05:47:39.502962 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:39.502882 2529 generic.go:356] "Generic (PLEG): container finished" podID="103bd762-0199-4273-a134-6b1528e1d975" containerID="c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3" exitCode=0 Jul 28 05:47:39.502962 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:39.502948 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerDied","Data":"c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3"} Jul 28 05:47:40.513650 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:40.513551 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerStarted","Data":"02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc"} Jul 28 05:47:40.513650 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:40.513620 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerStarted","Data":"3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f"} Jul 28 05:47:40.513650 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:40.513645 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerStarted","Data":"09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d"} Jul 28 05:47:40.514316 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:40.513666 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerStarted","Data":"43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15"} Jul 28 05:47:40.542841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:40.542761 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podStartSLOduration=13.542738822 podStartE2EDuration="13.542738822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:47:40.54217274 +0000 UTC m=+1523.474207901" watchObservedRunningTime="2026-07-28 05:47:40.542738822 +0000 UTC m=+1523.474773983" Jul 28 05:47:43.028650 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:43.028556 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:43.028650 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:43.028621 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:43.028650 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:43.028642 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:43.028650 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:43.028661 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:43.030768 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:43.030726 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused" Jul 28 05:47:43.030768 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:43.030743 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused" Jul 28 05:47:43.030961 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:43.030773 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jul 28 05:47:43.031069 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:43.031047 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jul 28 05:47:43.330008 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:43.329904 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 05:47:44.172808 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:44.172761 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 05:47:46.029874 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.029835 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.030324 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.029965 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.030324 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.030208 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.030612 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.030527 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.030612 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.030567 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.030612 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.030586 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.030842 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.030794 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.030937 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.030914 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.031045 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.030965 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.035119 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.035099 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.550885 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.550856 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:46.553939 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:46.553915 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:47:53.329498 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:53.329455 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 05:47:53.330074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:53.329601 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:47:54.172064 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:54.172019 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 05:47:54.172317 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:54.172166 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:47:57.464474 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.464450 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:47:57.599588 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.599552 2529 generic.go:356] "Generic (PLEG): container finished" podID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerID="6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad" exitCode=137 Jul 28 05:47:57.599588 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.599576 2529 generic.go:356] "Generic (PLEG): container finished" podID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerID="e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d" exitCode=137 Jul 28 05:47:57.599771 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.599621 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" Jul 28 05:47:57.599771 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.599629 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerDied","Data":"6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad"} Jul 28 05:47:57.599771 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.599674 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerDied","Data":"e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d"} Jul 28 05:47:57.599771 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.599690 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76" event={"ID":"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b","Type":"ContainerDied","Data":"d918f6a19debe2689fbae12afd3f2b8f01b4b9b00d6950bc3d2dc715d9614de3"} Jul 28 05:47:57.599771 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.599709 2529 scope.go:117] "RemoveContainer" containerID="4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5" Jul 28 05:47:57.614409 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.614381 2529 scope.go:117] "RemoveContainer" containerID="6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad" Jul 28 05:47:57.621922 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.621904 2529 scope.go:117] "RemoveContainer" containerID="4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680" Jul 28 05:47:57.629181 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.629165 2529 scope.go:117] "RemoveContainer" containerID="e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d" Jul 28 05:47:57.633220 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.633201 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-ui-tls\") pod \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " Jul 28 05:47:57.633311 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.633236 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-online-tls\") pod \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " Jul 28 05:47:57.633311 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.633275 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2b5f\" (UniqueName: \"kubernetes.io/projected/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-kube-api-access-n2b5f\") pod \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " Jul 28 05:47:57.633434 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.633316 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-offline-tls\") pod \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " Jul 28 05:47:57.633434 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.633370 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-registry-tls\") pod \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " Jul 28 05:47:57.633562 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.633444 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-feast-data\") pod \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\" (UID: \"a4ac4d92-6b36-40f6-8d24-8d49365c6a5b\") " Jul 28 05:47:57.634099 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.634070 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-feast-data" (OuterVolumeSpecName: "feast-data") pod "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" (UID: "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:47:57.635652 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.635607 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-kube-api-access-n2b5f" (OuterVolumeSpecName: "kube-api-access-n2b5f") pod "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" (UID: "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b"). InnerVolumeSpecName "kube-api-access-n2b5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:47:57.635652 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.635618 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" (UID: "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:47:57.635802 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.635756 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-online-tls" (OuterVolumeSpecName: "online-tls") pod "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" (UID: "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:47:57.636040 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.636013 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" (UID: "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:47:57.636119 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.636062 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" (UID: "a4ac4d92-6b36-40f6-8d24-8d49365c6a5b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:47:57.637485 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.637460 2529 scope.go:117] "RemoveContainer" containerID="9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb" Jul 28 05:47:57.647421 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.647402 2529 scope.go:117] "RemoveContainer" containerID="00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9" Jul 28 05:47:57.655047 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.655028 2529 scope.go:117] "RemoveContainer" containerID="4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5" Jul 28 05:47:57.655300 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:47:57.655279 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5\": container with ID starting with 4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5 not found: ID does not exist" containerID="4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5" Jul 28 05:47:57.655354 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.655313 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5"} err="failed to get container status \"4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5\": rpc error: code = NotFound desc = could not find container \"4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5\": container with ID starting with 4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5 not found: ID does not exist" Jul 28 05:47:57.655354 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.655334 2529 scope.go:117] "RemoveContainer" containerID="6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad" Jul 28 05:47:57.655593 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:47:57.655573 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad\": container with ID starting with 6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad not found: ID does not exist" containerID="6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad" Jul 28 05:47:57.655673 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.655600 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad"} err="failed to get container status \"6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad\": rpc error: code = NotFound desc = could not find container \"6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad\": container with ID starting with 6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad not found: ID does not exist" Jul 28 05:47:57.655673 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.655622 2529 scope.go:117] "RemoveContainer" containerID="4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680" Jul 28 05:47:57.655876 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:47:57.655857 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680\": container with ID starting with 4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680 not found: ID does not exist" containerID="4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680" Jul 28 05:47:57.655919 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.655880 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680"} err="failed to get container status \"4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680\": rpc error: code = NotFound desc = could not find container \"4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680\": container with ID starting with 4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680 not found: ID does not exist" Jul 28 05:47:57.655919 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.655896 2529 scope.go:117] "RemoveContainer" containerID="e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d" Jul 28 05:47:57.656114 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:47:57.656098 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d\": container with ID starting with e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d not found: ID does not exist" containerID="e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d" Jul 28 05:47:57.656171 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.656119 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d"} err="failed to get container status \"e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d\": rpc error: code = NotFound desc = could not find container \"e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d\": container with ID starting with e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d not found: ID does not exist" Jul 28 05:47:57.656171 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.656132 2529 scope.go:117] "RemoveContainer" containerID="9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb" Jul 28 05:47:57.656341 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:47:57.656320 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb\": container with ID starting with 9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb not found: ID does not exist" containerID="9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb" Jul 28 05:47:57.656403 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.656348 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb"} err="failed to get container status \"9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb\": rpc error: code = NotFound desc = could not find container \"9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb\": container with ID starting with 9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb not found: ID does not exist" Jul 28 05:47:57.656403 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.656369 2529 scope.go:117] "RemoveContainer" containerID="00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9" Jul 28 05:47:57.656609 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:47:57.656592 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9\": container with ID starting with 00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9 not found: ID does not exist" containerID="00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9" Jul 28 05:47:57.656679 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.656615 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9"} err="failed to get container status \"00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9\": rpc error: code = NotFound desc = could not find container \"00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9\": container with ID starting with 00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9 not found: ID does not exist" Jul 28 05:47:57.656679 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.656635 2529 scope.go:117] "RemoveContainer" containerID="4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5" Jul 28 05:47:57.656851 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.656832 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5"} err="failed to get container status \"4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5\": rpc error: code = NotFound desc = could not find container \"4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5\": container with ID starting with 4f5caca4870b52d8d6e32bf5870f5f3a4c9842049983934762969f2b4689b8f5 not found: ID does not exist" Jul 28 05:47:57.656918 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.656853 2529 scope.go:117] "RemoveContainer" containerID="6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad" Jul 28 05:47:57.657028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.657012 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad"} err="failed to get container status \"6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad\": rpc error: code = NotFound desc = could not find container \"6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad\": container with ID starting with 6e6d5579fa25c305656c2f4fd3f1d0d0483f0f9ced4f9e82b799c119ced358ad not found: ID does not exist" Jul 28 05:47:57.657093 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.657030 2529 scope.go:117] "RemoveContainer" containerID="4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680" Jul 28 05:47:57.657199 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.657183 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680"} err="failed to get container status \"4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680\": rpc error: code = NotFound desc = could not find container \"4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680\": container with ID starting with 4c4774fdec8ea7be278654b72e00b1607f0780d31e486502b2d04f9228c1b680 not found: ID does not exist" Jul 28 05:47:57.657261 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.657200 2529 scope.go:117] "RemoveContainer" containerID="e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d" Jul 28 05:47:57.657357 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.657339 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d"} err="failed to get container status \"e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d\": rpc error: code = NotFound desc = could not find container \"e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d\": container with ID starting with e4cad252287f273635aee45ea9f902fa19621f5b6abe875a0192842f2be2b81d not found: ID does not exist" Jul 28 05:47:57.657411 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.657360 2529 scope.go:117] "RemoveContainer" containerID="9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb" Jul 28 05:47:57.657568 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.657551 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb"} err="failed to get container status \"9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb\": rpc error: code = NotFound desc = could not find container \"9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb\": container with ID starting with 9b340eaf5f7df8c0dc1d26ce143f09fce8d530b89a9802852ed2241370313abb not found: ID does not exist" Jul 28 05:47:57.657631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.657569 2529 scope.go:117] "RemoveContainer" containerID="00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9" Jul 28 05:47:57.657767 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.657746 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9"} err="failed to get container status \"00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9\": rpc error: code = NotFound desc = could not find container \"00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9\": container with ID starting with 00180016212a1d53cc50b532cf87ddf773309d9760059e75355668b11bae7ff9 not found: ID does not exist" Jul 28 05:47:57.734394 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.734361 2529 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-registry-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:47:57.734394 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.734389 2529 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-feast-data\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:47:57.734394 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.734398 2529 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-ui-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:47:57.734634 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.734430 2529 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-online-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:47:57.734634 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.734441 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2b5f\" (UniqueName: \"kubernetes.io/projected/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-kube-api-access-n2b5f\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:47:57.734634 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.734451 2529 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b-offline-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:47:57.921608 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.921582 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76"] Jul 28 05:47:57.925042 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:57.925018 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-b2f76"] Jul 28 05:47:59.585674 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:47:59.585645 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" path="/var/lib/kubelet/pods/a4ac4d92-6b36-40f6-8d24-8d49365c6a5b/volumes" Jul 28 05:48:48.887405 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887369 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw"] Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887706 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="ui" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887717 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="ui" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887733 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="feast-apply" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887739 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="feast-apply" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887751 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="offline" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887758 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="offline" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887766 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="feast-init" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887771 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="feast-init" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887783 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="registry" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887790 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="registry" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887798 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="online" Jul 28 05:48:48.887841 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887803 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="online" Jul 28 05:48:48.888200 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887852 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="registry" Jul 28 05:48:48.888200 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887861 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="offline" Jul 28 05:48:48.888200 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887869 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="ui" Jul 28 05:48:48.888200 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.887877 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ac4d92-6b36-40f6-8d24-8d49365c6a5b" containerName="online" Jul 28 05:48:48.891028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.891013 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:48.893212 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.893176 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 28 05:48:48.893212 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.893205 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-v7wbb\"" Jul 28 05:48:48.893374 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.893178 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 28 05:48:48.893374 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.893234 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 28 05:48:48.893374 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.893180 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 28 05:48:48.893374 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.893307 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 28 05:48:48.893374 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.893215 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 28 05:48:48.903323 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.903298 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw"] Jul 28 05:48:48.977770 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.977736 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3a2ebc8c-6fc3-4494-a948-42201a9040e4-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:48.977938 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.977783 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3a2ebc8c-6fc3-4494-a948-42201a9040e4-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:48.977938 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.977812 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:48.977938 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.977852 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5vh\" (UniqueName: \"kubernetes.io/projected/3a2ebc8c-6fc3-4494-a948-42201a9040e4-kube-api-access-kg5vh\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:48.977938 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.977872 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:48.977938 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:48.977885 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.078599 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.078566 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5vh\" (UniqueName: \"kubernetes.io/projected/3a2ebc8c-6fc3-4494-a948-42201a9040e4-kube-api-access-kg5vh\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.078776 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.078607 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.078776 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.078626 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.078776 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.078684 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3a2ebc8c-6fc3-4494-a948-42201a9040e4-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.078776 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.078714 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3a2ebc8c-6fc3-4494-a948-42201a9040e4-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.078776 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.078741 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.079059 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:48:49.078782 2529 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 05:48:49.079059 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:48:49.078842 2529 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 05:48:49.079059 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:48:49.078862 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-online-tls podName:3a2ebc8c-6fc3-4494-a948-42201a9040e4 nodeName:}" failed. No retries permitted until 2026-07-28 05:48:49.578841624 +0000 UTC m=+1592.510876772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-online-tls") pod "feast-simple-feast-remote-setup-db6955c6c-t9gvw" (UID: "3a2ebc8c-6fc3-4494-a948-42201a9040e4") : secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 05:48:49.079059 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:48:49.078902 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-ui-tls podName:3a2ebc8c-6fc3-4494-a948-42201a9040e4 nodeName:}" failed. No retries permitted until 2026-07-28 05:48:49.578886177 +0000 UTC m=+1592.510921316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-ui-tls") pod "feast-simple-feast-remote-setup-db6955c6c-t9gvw" (UID: "3a2ebc8c-6fc3-4494-a948-42201a9040e4") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 05:48:49.079293 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.079110 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3a2ebc8c-6fc3-4494-a948-42201a9040e4-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.079489 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.079465 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3a2ebc8c-6fc3-4494-a948-42201a9040e4-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.081075 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.081053 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.086339 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.086315 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5vh\" (UniqueName: \"kubernetes.io/projected/3a2ebc8c-6fc3-4494-a948-42201a9040e4-kube-api-access-kg5vh\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.581909 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.581872 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.582105 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.581942 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.584368 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.584340 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.584502 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.584428 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-t9gvw\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.800429 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.800397 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:48:49.932421 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:49.932399 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw"] Jul 28 05:48:49.933714 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:48:49.933688 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a2ebc8c_6fc3_4494_a948_42201a9040e4.slice/crio-3011fc37209545096e8e1cbc5d228351c6d55f8203a1f17c7e9ea220b78dbbcd WatchSource:0}: Error finding container 3011fc37209545096e8e1cbc5d228351c6d55f8203a1f17c7e9ea220b78dbbcd: Status 404 returned error can't find the container with id 3011fc37209545096e8e1cbc5d228351c6d55f8203a1f17c7e9ea220b78dbbcd Jul 28 05:48:50.823732 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:50.823689 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerStarted","Data":"4530b71020d00c8a12c4e194c51be8dad37e69ca8309bc1a3e7e12f9b85e0da5"} Jul 28 05:48:50.823732 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:50.823737 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerStarted","Data":"3011fc37209545096e8e1cbc5d228351c6d55f8203a1f17c7e9ea220b78dbbcd"} Jul 28 05:48:54.841698 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:54.841663 2529 generic.go:356] "Generic (PLEG): container finished" podID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerID="4530b71020d00c8a12c4e194c51be8dad37e69ca8309bc1a3e7e12f9b85e0da5" exitCode=0 Jul 28 05:48:54.842059 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:54.841738 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerDied","Data":"4530b71020d00c8a12c4e194c51be8dad37e69ca8309bc1a3e7e12f9b85e0da5"} Jul 28 05:48:55.847690 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:48:55.847647 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerStarted","Data":"38c1319ea9d574316e1568e4e86c29f21a9a3c6a08fe300e97f12f89af016cb8"} Jul 28 05:49:00.869070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:00.868985 2529 generic.go:356] "Generic (PLEG): container finished" podID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerID="38c1319ea9d574316e1568e4e86c29f21a9a3c6a08fe300e97f12f89af016cb8" exitCode=0 Jul 28 05:49:00.869511 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:00.869058 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerDied","Data":"38c1319ea9d574316e1568e4e86c29f21a9a3c6a08fe300e97f12f89af016cb8"} Jul 28 05:49:01.877953 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:01.877913 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerStarted","Data":"0343fb703922d02f904c1b7cc10dfcd68c8133f2385821df9bafdda01d4d9c82"} Jul 28 05:49:01.878433 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:01.877968 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerStarted","Data":"da2a412314a53fd19e165c1309447bbc2436111a110f4a97d2e3af2893421c7c"} Jul 28 05:49:01.878433 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:01.877981 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerStarted","Data":"12492c5e648b2df7b6ab4cc57130b6e0a4ec1f2f8aa57dc373be00280631d618"} Jul 28 05:49:01.903237 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:01.903179 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podStartSLOduration=13.903160063 podStartE2EDuration="13.903160063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:49:01.901605538 +0000 UTC m=+1604.833640699" watchObservedRunningTime="2026-07-28 05:49:01.903160063 +0000 UTC m=+1604.835195224" Jul 28 05:49:04.800760 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:04.800719 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:04.801306 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:04.801277 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:04.801306 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:04.801310 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:04.802703 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:04.802669 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Jul 28 05:49:04.802814 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:04.802715 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused" Jul 28 05:49:04.802814 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:04.802718 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Jul 28 05:49:07.802338 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:07.802307 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:07.802857 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:07.802439 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:07.802931 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:07.802872 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:07.802931 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:07.802910 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:07.803118 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:07.803099 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:07.803366 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:07.803349 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:07.809010 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:07.808989 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:07.907907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:07.907877 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:49:07.912570 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:49:07.912547 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:50:10.254874 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:10.254842 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw"] Jul 28 05:50:10.255511 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:10.255080 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="online" containerID="cri-o://12492c5e648b2df7b6ab4cc57130b6e0a4ec1f2f8aa57dc373be00280631d618" gracePeriod=30 Jul 28 05:50:10.255511 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:10.255117 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="offline" containerID="cri-o://da2a412314a53fd19e165c1309447bbc2436111a110f4a97d2e3af2893421c7c" gracePeriod=30 Jul 28 05:50:10.255511 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:10.255119 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="ui" containerID="cri-o://0343fb703922d02f904c1b7cc10dfcd68c8133f2385821df9bafdda01d4d9c82" gracePeriod=30 Jul 28 05:50:10.391180 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:10.391149 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt"] Jul 28 05:50:10.391431 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:10.391399 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="registry" containerID="cri-o://43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15" gracePeriod=30 Jul 28 05:50:10.391584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:10.391434 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="offline" containerID="cri-o://3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f" gracePeriod=30 Jul 28 05:50:10.391584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:10.391485 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="ui" containerID="cri-o://02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc" gracePeriod=30 Jul 28 05:50:10.391584 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:10.391442 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="online" containerID="cri-o://09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d" gracePeriod=30 Jul 28 05:50:11.182674 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:11.182644 2529 generic.go:356] "Generic (PLEG): container finished" podID="103bd762-0199-4273-a134-6b1528e1d975" containerID="02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc" exitCode=0 Jul 28 05:50:11.182674 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:11.182668 2529 generic.go:356] "Generic (PLEG): container finished" podID="103bd762-0199-4273-a134-6b1528e1d975" containerID="09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d" exitCode=0 Jul 28 05:50:11.182900 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:11.182712 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerDied","Data":"02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc"} Jul 28 05:50:11.182900 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:11.182765 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerDied","Data":"09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d"} Jul 28 05:50:11.185481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:11.185459 2529 generic.go:356] "Generic (PLEG): container finished" podID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerID="0343fb703922d02f904c1b7cc10dfcd68c8133f2385821df9bafdda01d4d9c82" exitCode=0 Jul 28 05:50:11.185481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:11.185476 2529 generic.go:356] "Generic (PLEG): container finished" podID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerID="12492c5e648b2df7b6ab4cc57130b6e0a4ec1f2f8aa57dc373be00280631d618" exitCode=0 Jul 28 05:50:11.185660 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:11.185545 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerDied","Data":"0343fb703922d02f904c1b7cc10dfcd68c8133f2385821df9bafdda01d4d9c82"} Jul 28 05:50:11.185660 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:11.185576 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerDied","Data":"12492c5e648b2df7b6ab4cc57130b6e0a4ec1f2f8aa57dc373be00280631d618"} Jul 28 05:50:16.031260 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:16.031211 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jul 28 05:50:16.551195 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:16.551147 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jul 28 05:50:17.803944 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:17.803899 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Jul 28 05:50:17.908612 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:17.908562 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Jul 28 05:50:26.031217 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:26.031155 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jul 28 05:50:26.551565 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:26.551502 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jul 28 05:50:27.803189 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:27.803145 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Jul 28 05:50:27.908379 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:27.908340 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Jul 28 05:50:36.031451 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:36.031389 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jul 28 05:50:36.031932 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:36.031612 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:50:36.551007 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:36.550963 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jul 28 05:50:36.551212 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:36.551068 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:50:37.803932 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:37.803884 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Jul 28 05:50:37.804397 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:37.804023 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:50:37.908480 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:37.908434 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Jul 28 05:50:37.908663 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:37.908597 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:50:40.307940 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.307910 2529 generic.go:356] "Generic (PLEG): container finished" podID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerID="da2a412314a53fd19e165c1309447bbc2436111a110f4a97d2e3af2893421c7c" exitCode=137 Jul 28 05:50:40.308326 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.307988 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerDied","Data":"da2a412314a53fd19e165c1309447bbc2436111a110f4a97d2e3af2893421c7c"} Jul 28 05:50:40.509789 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.509727 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:50:40.541696 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.541666 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-ui-tls\") pod \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " Jul 28 05:50:40.541857 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.541739 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3a2ebc8c-6fc3-4494-a948-42201a9040e4-registry-tls\") pod \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " Jul 28 05:50:40.541927 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.541849 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3a2ebc8c-6fc3-4494-a948-42201a9040e4-feast-data\") pod \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " Jul 28 05:50:40.541927 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.541892 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-offline-tls\") pod \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " Jul 28 05:50:40.542042 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.541995 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg5vh\" (UniqueName: \"kubernetes.io/projected/3a2ebc8c-6fc3-4494-a948-42201a9040e4-kube-api-access-kg5vh\") pod \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " Jul 28 05:50:40.542101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.542043 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-online-tls\") pod \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\" (UID: \"3a2ebc8c-6fc3-4494-a948-42201a9040e4\") " Jul 28 05:50:40.542101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.542082 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a2ebc8c-6fc3-4494-a948-42201a9040e4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3a2ebc8c-6fc3-4494-a948-42201a9040e4" (UID: "3a2ebc8c-6fc3-4494-a948-42201a9040e4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:50:40.542285 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.542260 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2ebc8c-6fc3-4494-a948-42201a9040e4-feast-data" (OuterVolumeSpecName: "feast-data") pod "3a2ebc8c-6fc3-4494-a948-42201a9040e4" (UID: "3a2ebc8c-6fc3-4494-a948-42201a9040e4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:50:40.542343 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.542276 2529 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3a2ebc8c-6fc3-4494-a948-42201a9040e4-registry-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:40.544067 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.544039 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2ebc8c-6fc3-4494-a948-42201a9040e4-kube-api-access-kg5vh" (OuterVolumeSpecName: "kube-api-access-kg5vh") pod "3a2ebc8c-6fc3-4494-a948-42201a9040e4" (UID: "3a2ebc8c-6fc3-4494-a948-42201a9040e4"). InnerVolumeSpecName "kube-api-access-kg5vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:50:40.544164 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.544118 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "3a2ebc8c-6fc3-4494-a948-42201a9040e4" (UID: "3a2ebc8c-6fc3-4494-a948-42201a9040e4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:50:40.544213 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.544161 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-online-tls" (OuterVolumeSpecName: "online-tls") pod "3a2ebc8c-6fc3-4494-a948-42201a9040e4" (UID: "3a2ebc8c-6fc3-4494-a948-42201a9040e4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:50:40.544409 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.544394 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "3a2ebc8c-6fc3-4494-a948-42201a9040e4" (UID: "3a2ebc8c-6fc3-4494-a948-42201a9040e4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:50:40.643314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.643272 2529 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3a2ebc8c-6fc3-4494-a948-42201a9040e4-feast-data\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:40.643314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.643306 2529 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-offline-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:40.643314 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.643317 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5vh\" (UniqueName: \"kubernetes.io/projected/3a2ebc8c-6fc3-4494-a948-42201a9040e4-kube-api-access-kg5vh\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:40.643569 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.643347 2529 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-online-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:40.643569 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:40.643358 2529 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3a2ebc8c-6fc3-4494-a948-42201a9040e4-ui-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:41.037090 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.037064 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:50:41.147726 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.147693 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-offline-tls\") pod \"103bd762-0199-4273-a134-6b1528e1d975\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " Jul 28 05:50:41.147726 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.147731 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/103bd762-0199-4273-a134-6b1528e1d975-feast-data\") pod \"103bd762-0199-4273-a134-6b1528e1d975\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " Jul 28 05:50:41.147936 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.147764 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-registry-tls\") pod \"103bd762-0199-4273-a134-6b1528e1d975\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " Jul 28 05:50:41.147936 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.147793 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-online-tls\") pod \"103bd762-0199-4273-a134-6b1528e1d975\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " Jul 28 05:50:41.147936 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.147827 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7v2n\" (UniqueName: \"kubernetes.io/projected/103bd762-0199-4273-a134-6b1528e1d975-kube-api-access-f7v2n\") pod \"103bd762-0199-4273-a134-6b1528e1d975\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " Jul 28 05:50:41.147936 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.147853 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-ui-tls\") pod \"103bd762-0199-4273-a134-6b1528e1d975\" (UID: \"103bd762-0199-4273-a134-6b1528e1d975\") " Jul 28 05:50:41.148360 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.148327 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bd762-0199-4273-a134-6b1528e1d975-feast-data" (OuterVolumeSpecName: "feast-data") pod "103bd762-0199-4273-a134-6b1528e1d975" (UID: "103bd762-0199-4273-a134-6b1528e1d975"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:50:41.149980 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.149938 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103bd762-0199-4273-a134-6b1528e1d975-kube-api-access-f7v2n" (OuterVolumeSpecName: "kube-api-access-f7v2n") pod "103bd762-0199-4273-a134-6b1528e1d975" (UID: "103bd762-0199-4273-a134-6b1528e1d975"). InnerVolumeSpecName "kube-api-access-f7v2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:50:41.150077 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.149997 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "103bd762-0199-4273-a134-6b1528e1d975" (UID: "103bd762-0199-4273-a134-6b1528e1d975"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:50:41.150077 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.150014 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "103bd762-0199-4273-a134-6b1528e1d975" (UID: "103bd762-0199-4273-a134-6b1528e1d975"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:50:41.150278 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.150251 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-online-tls" (OuterVolumeSpecName: "online-tls") pod "103bd762-0199-4273-a134-6b1528e1d975" (UID: "103bd762-0199-4273-a134-6b1528e1d975"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:50:41.150278 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.150269 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "103bd762-0199-4273-a134-6b1528e1d975" (UID: "103bd762-0199-4273-a134-6b1528e1d975"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:50:41.248902 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.248827 2529 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-registry-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:41.248902 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.248854 2529 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-online-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:41.248902 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.248865 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7v2n\" (UniqueName: \"kubernetes.io/projected/103bd762-0199-4273-a134-6b1528e1d975-kube-api-access-f7v2n\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:41.248902 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.248874 2529 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-ui-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:41.248902 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.248882 2529 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/103bd762-0199-4273-a134-6b1528e1d975-offline-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:41.248902 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.248891 2529 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/103bd762-0199-4273-a134-6b1528e1d975-feast-data\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:50:41.315441 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.315412 2529 generic.go:356] "Generic (PLEG): container finished" podID="103bd762-0199-4273-a134-6b1528e1d975" containerID="3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f" exitCode=137 Jul 28 05:50:41.315441 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.315432 2529 generic.go:356] "Generic (PLEG): container finished" podID="103bd762-0199-4273-a134-6b1528e1d975" containerID="43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15" exitCode=137 Jul 28 05:50:41.315940 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.315479 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" Jul 28 05:50:41.315940 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.315491 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerDied","Data":"3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f"} Jul 28 05:50:41.315940 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.315557 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerDied","Data":"43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15"} Jul 28 05:50:41.315940 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.315576 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt" event={"ID":"103bd762-0199-4273-a134-6b1528e1d975","Type":"ContainerDied","Data":"e4cf9e37c0185b64dba58d55408eada9ccbebf5b194c3ba136c58b5db660008e"} Jul 28 05:50:41.315940 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.315593 2529 scope.go:117] "RemoveContainer" containerID="02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc" Jul 28 05:50:41.318259 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.318232 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" event={"ID":"3a2ebc8c-6fc3-4494-a948-42201a9040e4","Type":"ContainerDied","Data":"3011fc37209545096e8e1cbc5d228351c6d55f8203a1f17c7e9ea220b78dbbcd"} Jul 28 05:50:41.318259 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.318257 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw" Jul 28 05:50:41.325351 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.325337 2529 scope.go:117] "RemoveContainer" containerID="3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f" Jul 28 05:50:41.332994 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.332974 2529 scope.go:117] "RemoveContainer" containerID="09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d" Jul 28 05:50:41.338138 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.338119 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt"] Jul 28 05:50:41.341000 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.340977 2529 scope.go:117] "RemoveContainer" containerID="43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15" Jul 28 05:50:41.344103 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.344083 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-rl8dt"] Jul 28 05:50:41.348912 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.348882 2529 scope.go:117] "RemoveContainer" containerID="c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3" Jul 28 05:50:41.356045 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.355993 2529 scope.go:117] "RemoveContainer" containerID="0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226" Jul 28 05:50:41.358964 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.358945 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw"] Jul 28 05:50:41.360223 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.360205 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-t9gvw"] Jul 28 05:50:41.364727 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.364711 2529 scope.go:117] "RemoveContainer" containerID="02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc" Jul 28 05:50:41.364982 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:50:41.364963 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc\": container with ID starting with 02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc not found: ID does not exist" containerID="02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc" Jul 28 05:50:41.365028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.364990 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc"} err="failed to get container status \"02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc\": rpc error: code = NotFound desc = could not find container \"02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc\": container with ID starting with 02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc not found: ID does not exist" Jul 28 05:50:41.365028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.365008 2529 scope.go:117] "RemoveContainer" containerID="3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f" Jul 28 05:50:41.365220 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:50:41.365202 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f\": container with ID starting with 3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f not found: ID does not exist" containerID="3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f" Jul 28 05:50:41.365263 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.365228 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f"} err="failed to get container status \"3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f\": rpc error: code = NotFound desc = could not find container \"3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f\": container with ID starting with 3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f not found: ID does not exist" Jul 28 05:50:41.365263 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.365244 2529 scope.go:117] "RemoveContainer" containerID="09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d" Jul 28 05:50:41.365437 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:50:41.365421 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d\": container with ID starting with 09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d not found: ID does not exist" containerID="09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d" Jul 28 05:50:41.365477 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.365439 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d"} err="failed to get container status \"09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d\": rpc error: code = NotFound desc = could not find container \"09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d\": container with ID starting with 09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d not found: ID does not exist" Jul 28 05:50:41.365477 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.365451 2529 scope.go:117] "RemoveContainer" containerID="43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15" Jul 28 05:50:41.365657 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:50:41.365641 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15\": container with ID starting with 43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15 not found: ID does not exist" containerID="43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15" Jul 28 05:50:41.365701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.365661 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15"} err="failed to get container status \"43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15\": rpc error: code = NotFound desc = could not find container \"43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15\": container with ID starting with 43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15 not found: ID does not exist" Jul 28 05:50:41.365701 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.365672 2529 scope.go:117] "RemoveContainer" containerID="c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3" Jul 28 05:50:41.365882 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:50:41.365864 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3\": container with ID starting with c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3 not found: ID does not exist" containerID="c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3" Jul 28 05:50:41.365922 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.365887 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3"} err="failed to get container status \"c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3\": rpc error: code = NotFound desc = could not find container \"c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3\": container with ID starting with c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3 not found: ID does not exist" Jul 28 05:50:41.365922 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.365901 2529 scope.go:117] "RemoveContainer" containerID="0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226" Jul 28 05:50:41.366094 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:50:41.366080 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226\": container with ID starting with 0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226 not found: ID does not exist" containerID="0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226" Jul 28 05:50:41.366137 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366097 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226"} err="failed to get container status \"0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226\": rpc error: code = NotFound desc = could not find container \"0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226\": container with ID starting with 0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226 not found: ID does not exist" Jul 28 05:50:41.366137 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366108 2529 scope.go:117] "RemoveContainer" containerID="02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc" Jul 28 05:50:41.366346 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366328 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc"} err="failed to get container status \"02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc\": rpc error: code = NotFound desc = could not find container \"02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc\": container with ID starting with 02960a15fd56659edd18d79979b5215300b3b0891e48123bcdbb90c90fbfdedc not found: ID does not exist" Jul 28 05:50:41.366397 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366346 2529 scope.go:117] "RemoveContainer" containerID="3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f" Jul 28 05:50:41.366545 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366529 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f"} err="failed to get container status \"3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f\": rpc error: code = NotFound desc = could not find container \"3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f\": container with ID starting with 3b9d68295406751fe872eb8605300a588fbcea60b016dad6fcd0dd9093d0f72f not found: ID does not exist" Jul 28 05:50:41.366595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366544 2529 scope.go:117] "RemoveContainer" containerID="09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d" Jul 28 05:50:41.366723 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366706 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d"} err="failed to get container status \"09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d\": rpc error: code = NotFound desc = could not find container \"09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d\": container with ID starting with 09dee92bf03099295abba490f9c65c4aee58c181dd377615d0a1d24c332c9d3d not found: ID does not exist" Jul 28 05:50:41.366760 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366723 2529 scope.go:117] "RemoveContainer" containerID="43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15" Jul 28 05:50:41.366907 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366890 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15"} err="failed to get container status \"43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15\": rpc error: code = NotFound desc = could not find container \"43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15\": container with ID starting with 43b764f5e5a8aa7bd6384e1433cc45ae9e66b702ace04d744d2b8abc5468ea15 not found: ID does not exist" Jul 28 05:50:41.366950 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.366907 2529 scope.go:117] "RemoveContainer" containerID="c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3" Jul 28 05:50:41.367112 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.367096 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3"} err="failed to get container status \"c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3\": rpc error: code = NotFound desc = could not find container \"c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3\": container with ID starting with c3df7ea013d4971efb1f6f6e54d21b811cc3cb46c8600f7678ff218059e537e3 not found: ID does not exist" Jul 28 05:50:41.367112 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.367112 2529 scope.go:117] "RemoveContainer" containerID="0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226" Jul 28 05:50:41.367274 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.367257 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226"} err="failed to get container status \"0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226\": rpc error: code = NotFound desc = could not find container \"0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226\": container with ID starting with 0942b4aed48f60003d53be53fe069d5346ae6fb6d4e8f76b90b3b24dc61d2226 not found: ID does not exist" Jul 28 05:50:41.367316 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.367274 2529 scope.go:117] "RemoveContainer" containerID="0343fb703922d02f904c1b7cc10dfcd68c8133f2385821df9bafdda01d4d9c82" Jul 28 05:50:41.374625 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.374608 2529 scope.go:117] "RemoveContainer" containerID="da2a412314a53fd19e165c1309447bbc2436111a110f4a97d2e3af2893421c7c" Jul 28 05:50:41.381665 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.381650 2529 scope.go:117] "RemoveContainer" containerID="12492c5e648b2df7b6ab4cc57130b6e0a4ec1f2f8aa57dc373be00280631d618" Jul 28 05:50:41.388583 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.388567 2529 scope.go:117] "RemoveContainer" containerID="38c1319ea9d574316e1568e4e86c29f21a9a3c6a08fe300e97f12f89af016cb8" Jul 28 05:50:41.395193 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.395178 2529 scope.go:117] "RemoveContainer" containerID="4530b71020d00c8a12c4e194c51be8dad37e69ca8309bc1a3e7e12f9b85e0da5" Jul 28 05:50:41.585931 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.585854 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103bd762-0199-4273-a134-6b1528e1d975" path="/var/lib/kubelet/pods/103bd762-0199-4273-a134-6b1528e1d975/volumes" Jul 28 05:50:41.586498 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:50:41.586484 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" path="/var/lib/kubelet/pods/3a2ebc8c-6fc3-4494-a948-42201a9040e4/volumes" Jul 28 05:51:20.079071 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079038 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j"] Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079347 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="online" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079361 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="online" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079375 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="offline" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079383 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="offline" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079403 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="registry" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079410 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="registry" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079420 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="feast-init" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079426 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="feast-init" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079432 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="ui" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079437 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="ui" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079449 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="ui" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079455 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="ui" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079462 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="feast-apply" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079468 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="feast-apply" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079473 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="feast-init" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079478 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="feast-init" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079484 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="feast-apply" Jul 28 05:51:20.079481 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079489 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="feast-apply" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079497 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="online" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079502 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="online" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079507 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="offline" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079524 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="offline" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079577 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="offline" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079586 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="offline" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079592 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="ui" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079598 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="online" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079604 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="ui" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079610 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="103bd762-0199-4273-a134-6b1528e1d975" containerName="registry" Jul 28 05:51:20.080074 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.079617 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2ebc8c-6fc3-4494-a948-42201a9040e4" containerName="online" Jul 28 05:51:20.082981 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.082964 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.085192 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.084999 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 28 05:51:20.085192 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.085015 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 05:51:20.085192 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.085053 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 28 05:51:20.085192 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.085072 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 28 05:51:20.085192 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.085110 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 05:51:20.085192 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.085025 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 28 05:51:20.085192 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.085163 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-znqdr\"" Jul 28 05:51:20.093029 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.093007 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j"] Jul 28 05:51:20.177844 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.177807 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.177844 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.177847 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmq6\" (UniqueName: \"kubernetes.io/projected/2d258b9f-85be-47cc-a145-93a15863a0ee-kube-api-access-zxmq6\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.178070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.177918 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.178070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.177944 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d258b9f-85be-47cc-a145-93a15863a0ee-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.178070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.177992 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.178070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.178018 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.278947 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.278910 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.279131 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.278966 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.279131 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.279049 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.279131 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.279076 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmq6\" (UniqueName: \"kubernetes.io/projected/2d258b9f-85be-47cc-a145-93a15863a0ee-kube-api-access-zxmq6\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.279131 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.279121 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.279343 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.279141 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d258b9f-85be-47cc-a145-93a15863a0ee-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.279343 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:51:20.279190 2529 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 28 05:51:20.279343 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:51:20.279281 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-registry-tls podName:2d258b9f-85be-47cc-a145-93a15863a0ee nodeName:}" failed. No retries permitted until 2026-07-28 05:51:20.779255949 +0000 UTC m=+1743.711291092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-registry-tls") pod "feast-simple-feast-setup-777d6cc998-2nb7j" (UID: "2d258b9f-85be-47cc-a145-93a15863a0ee") : secret "feast-simple-feast-setup-registry-tls" not found Jul 28 05:51:20.279546 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.279502 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d258b9f-85be-47cc-a145-93a15863a0ee-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.281560 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.281535 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.281680 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.281569 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.281680 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.281644 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.289030 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.289007 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmq6\" (UniqueName: \"kubernetes.io/projected/2d258b9f-85be-47cc-a145-93a15863a0ee-kube-api-access-zxmq6\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.784143 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.784083 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.786468 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.786438 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-2nb7j\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:20.999412 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:20.999374 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:21.131368 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:21.131345 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j"] Jul 28 05:51:21.133154 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:51:21.133123 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d258b9f_85be_47cc_a145_93a15863a0ee.slice/crio-8e881df972f352b931913218656d52528629cd0389c4d528ac8e2d42e1ad680a WatchSource:0}: Error finding container 8e881df972f352b931913218656d52528629cd0389c4d528ac8e2d42e1ad680a: Status 404 returned error can't find the container with id 8e881df972f352b931913218656d52528629cd0389c4d528ac8e2d42e1ad680a Jul 28 05:51:21.483668 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:21.483632 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerStarted","Data":"64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb"} Jul 28 05:51:21.483815 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:21.483679 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerStarted","Data":"8e881df972f352b931913218656d52528629cd0389c4d528ac8e2d42e1ad680a"} Jul 28 05:51:26.508855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:26.508817 2529 generic.go:356] "Generic (PLEG): container finished" podID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerID="64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb" exitCode=0 Jul 28 05:51:26.509270 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:26.508891 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerDied","Data":"64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb"} Jul 28 05:51:27.514248 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:27.514214 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerStarted","Data":"1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1"} Jul 28 05:51:32.534783 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:32.534747 2529 generic.go:356] "Generic (PLEG): container finished" podID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerID="1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1" exitCode=0 Jul 28 05:51:32.535148 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:32.534816 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerDied","Data":"1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1"} Jul 28 05:51:33.544888 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:33.544849 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerStarted","Data":"a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda"} Jul 28 05:51:33.545402 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:33.544916 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerStarted","Data":"99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd"} Jul 28 05:51:33.545402 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:33.544938 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerStarted","Data":"1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f"} Jul 28 05:51:33.545402 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:33.544956 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerStarted","Data":"5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0"} Jul 28 05:51:33.571021 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:33.570952 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podStartSLOduration=13.570929356 podStartE2EDuration="13.570929356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:51:33.567501869 +0000 UTC m=+1756.499537030" watchObservedRunningTime="2026-07-28 05:51:33.570929356 +0000 UTC m=+1756.502964516" Jul 28 05:51:36.000474 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:36.000407 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:36.000474 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:36.000479 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:36.001035 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:36.000499 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:36.001035 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:36.000535 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:36.002556 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:36.002460 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Jul 28 05:51:36.002556 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:36.002493 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Jul 28 05:51:36.002763 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:36.002613 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Jul 28 05:51:36.002763 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:36.002626 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 05:51:39.001206 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.001171 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.001830 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.001807 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.001918 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.001845 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.001990 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.001976 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.002051 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.002030 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.006382 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.006365 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.588997 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.588970 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.589202 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.589062 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.589202 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.589082 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.589415 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.589392 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.589595 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.589575 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:51:39.592086 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:51:39.592070 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:52:17.547270 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:17.547235 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:52:17.552039 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:17.552018 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:52:41.181231 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.181197 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j"] Jul 28 05:52:41.181777 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.181542 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="registry" containerID="cri-o://5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0" gracePeriod=30 Jul 28 05:52:41.181777 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.181614 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="online" containerID="cri-o://1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f" gracePeriod=30 Jul 28 05:52:41.181777 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.181629 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="ui" containerID="cri-o://a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda" gracePeriod=30 Jul 28 05:52:41.181777 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.181619 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="offline" containerID="cri-o://99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd" gracePeriod=30 Jul 28 05:52:41.485209 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.485115 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv"] Jul 28 05:52:41.489485 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.489455 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.491618 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.491593 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-48zxg\"" Jul 28 05:52:41.501855 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.501827 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv"] Jul 28 05:52:41.583044 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.583010 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.583227 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.583113 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.583227 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.583169 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.583227 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.583207 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qfw\" (UniqueName: \"kubernetes.io/projected/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-kube-api-access-v9qfw\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.583353 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.583256 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.583353 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.583292 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.683833 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.683790 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.684048 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.683861 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.684048 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.683940 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.684048 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:52:41.683958 2529 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 28 05:52:41.684048 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.683992 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.684048 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:52:41.684033 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-registry-tls podName:e2a6c7d9-341d-4ff5-8997-27993fa0d2ac nodeName:}" failed. No retries permitted until 2026-07-28 05:52:42.184010677 +0000 UTC m=+1825.116045817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-registry-tls") pod "feast-simple-feast-setup-777d6cc998-v9rbv" (UID: "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac") : secret "feast-simple-feast-setup-registry-tls" not found Jul 28 05:52:41.684541 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.684073 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.684541 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:52:41.684085 2529 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 28 05:52:41.684541 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.684110 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qfw\" (UniqueName: \"kubernetes.io/projected/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-kube-api-access-v9qfw\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.684541 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:52:41.684138 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls podName:e2a6c7d9-341d-4ff5-8997-27993fa0d2ac nodeName:}" failed. No retries permitted until 2026-07-28 05:52:42.184120274 +0000 UTC m=+1825.116155412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls") pod "feast-simple-feast-setup-777d6cc998-v9rbv" (UID: "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac") : secret "feast-simple-feast-setup-offline-tls" not found Jul 28 05:52:41.684541 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.684315 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.686669 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.686630 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.686669 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.686648 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.695187 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.695150 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qfw\" (UniqueName: \"kubernetes.io/projected/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-kube-api-access-v9qfw\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:41.837171 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.837141 2529 generic.go:356] "Generic (PLEG): container finished" podID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerID="a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda" exitCode=0 Jul 28 05:52:41.837339 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:41.837220 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerDied","Data":"a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda"} Jul 28 05:52:42.188393 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:42.188357 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:42.188795 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:42.188418 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:42.188795 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:52:42.188508 2529 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 28 05:52:42.188795 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:52:42.188588 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls podName:e2a6c7d9-341d-4ff5-8997-27993fa0d2ac nodeName:}" failed. No retries permitted until 2026-07-28 05:52:43.188571794 +0000 UTC m=+1826.120606947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls") pod "feast-simple-feast-setup-777d6cc998-v9rbv" (UID: "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac") : secret "feast-simple-feast-setup-offline-tls" not found Jul 28 05:52:42.190788 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:42.190769 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:42.844561 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:42.844504 2529 generic.go:356] "Generic (PLEG): container finished" podID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerID="1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f" exitCode=0 Jul 28 05:52:42.844561 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:42.844554 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerDied","Data":"1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f"} Jul 28 05:52:43.197409 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:43.197321 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:43.199677 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:43.199657 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-v9rbv\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:43.302262 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:43.302228 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:43.435457 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:43.435425 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv"] Jul 28 05:52:43.437841 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:52:43.437807 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2a6c7d9_341d_4ff5_8997_27993fa0d2ac.slice/crio-8665d9989afe0d502e3d13d1f8c2cad391159646c6c9f9780026e6f16d536c12 WatchSource:0}: Error finding container 8665d9989afe0d502e3d13d1f8c2cad391159646c6c9f9780026e6f16d536c12: Status 404 returned error can't find the container with id 8665d9989afe0d502e3d13d1f8c2cad391159646c6c9f9780026e6f16d536c12 Jul 28 05:52:43.850170 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:43.850138 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerStarted","Data":"58999fc65befec5a3f486d559382a7c2643749e30f7c0daf0a0730fb77ace6ef"} Jul 28 05:52:43.850330 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:43.850178 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerStarted","Data":"8665d9989afe0d502e3d13d1f8c2cad391159646c6c9f9780026e6f16d536c12"} Jul 28 05:52:48.874245 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:48.874210 2529 generic.go:356] "Generic (PLEG): container finished" podID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerID="58999fc65befec5a3f486d559382a7c2643749e30f7c0daf0a0730fb77ace6ef" exitCode=0 Jul 28 05:52:48.874643 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:48.874287 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerDied","Data":"58999fc65befec5a3f486d559382a7c2643749e30f7c0daf0a0730fb77ace6ef"} Jul 28 05:52:49.590055 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:49.590007 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Jul 28 05:52:49.590055 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:49.590011 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 05:52:49.879849 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:49.879816 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerStarted","Data":"955ae72e69c083a0473a9b27b74acff83596eddabe52fcbe04b256d7f2ed8872"} Jul 28 05:52:54.906177 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:54.906144 2529 generic.go:356] "Generic (PLEG): container finished" podID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerID="955ae72e69c083a0473a9b27b74acff83596eddabe52fcbe04b256d7f2ed8872" exitCode=0 Jul 28 05:52:54.906562 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:54.906216 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerDied","Data":"955ae72e69c083a0473a9b27b74acff83596eddabe52fcbe04b256d7f2ed8872"} Jul 28 05:52:55.916703 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:55.916652 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerStarted","Data":"d7beb44f2b3a237f901c49811c60c303b38612c4aa68ac52cb6817ac592538bc"} Jul 28 05:52:55.917209 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:55.916716 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerStarted","Data":"a443f1258f3bee295f6c458714f1769f23f186600e62ff2aa6471947f62a850e"} Jul 28 05:52:55.917209 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:55.916740 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerStarted","Data":"8e07a5e673032c5398104b046803c53d30cce188fa771841c1dd99bdc54c7b0f"} Jul 28 05:52:55.917209 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:55.916759 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerStarted","Data":"224144a347c4f3cb8b2c4fd364c2ef93a37a53d1fc7616becad381be96097d8b"} Jul 28 05:52:55.939990 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:55.939927 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podStartSLOduration=14.939877694 podStartE2EDuration="14.939877694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:52:55.938581484 +0000 UTC m=+1838.870616646" watchObservedRunningTime="2026-07-28 05:52:55.939877694 +0000 UTC m=+1838.871912855" Jul 28 05:52:58.303367 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:58.303206 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:58.303367 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:58.303346 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:58.303367 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:58.303371 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:58.304007 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:58.303407 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:52:58.305585 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:58.305546 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.28:8816: connect: connection refused" Jul 28 05:52:58.305735 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:58.305581 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" Jul 28 05:52:58.305735 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:58.305682 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Jul 28 05:52:58.305931 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:58.305897 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused" Jul 28 05:52:59.589569 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:59.589480 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Jul 28 05:52:59.590057 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:52:59.589824 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 05:53:01.303251 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.303217 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.303719 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.303359 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.303798 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.303759 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.303950 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.303935 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.304030 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.303964 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.304030 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.304021 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.304295 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.304275 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.308976 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.308955 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.950216 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.950173 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.950423 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.950225 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.950860 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.950835 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:01.953317 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:01.953295 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:53:09.589378 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:09.589332 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Jul 28 05:53:09.589825 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:09.589353 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 05:53:09.591198 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:09.591175 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:53:09.591316 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:09.591233 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:53:11.839762 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.839738 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:53:11.859912 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.859888 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxmq6\" (UniqueName: \"kubernetes.io/projected/2d258b9f-85be-47cc-a145-93a15863a0ee-kube-api-access-zxmq6\") pod \"2d258b9f-85be-47cc-a145-93a15863a0ee\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " Jul 28 05:53:11.860028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.859942 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-offline-tls\") pod \"2d258b9f-85be-47cc-a145-93a15863a0ee\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " Jul 28 05:53:11.860028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.859969 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d258b9f-85be-47cc-a145-93a15863a0ee-feast-data\") pod \"2d258b9f-85be-47cc-a145-93a15863a0ee\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " Jul 28 05:53:11.860028 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.860018 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-ui-tls\") pod \"2d258b9f-85be-47cc-a145-93a15863a0ee\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " Jul 28 05:53:11.860141 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.860057 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-registry-tls\") pod \"2d258b9f-85be-47cc-a145-93a15863a0ee\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " Jul 28 05:53:11.860141 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.860081 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-online-tls\") pod \"2d258b9f-85be-47cc-a145-93a15863a0ee\" (UID: \"2d258b9f-85be-47cc-a145-93a15863a0ee\") " Jul 28 05:53:11.860587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.860560 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d258b9f-85be-47cc-a145-93a15863a0ee-feast-data" (OuterVolumeSpecName: "feast-data") pod "2d258b9f-85be-47cc-a145-93a15863a0ee" (UID: "2d258b9f-85be-47cc-a145-93a15863a0ee"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:53:11.862351 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.862330 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2d258b9f-85be-47cc-a145-93a15863a0ee" (UID: "2d258b9f-85be-47cc-a145-93a15863a0ee"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:53:11.862553 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.862507 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2d258b9f-85be-47cc-a145-93a15863a0ee" (UID: "2d258b9f-85be-47cc-a145-93a15863a0ee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:53:11.862630 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.862544 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2d258b9f-85be-47cc-a145-93a15863a0ee" (UID: "2d258b9f-85be-47cc-a145-93a15863a0ee"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:53:11.862678 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.862640 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-online-tls" (OuterVolumeSpecName: "online-tls") pod "2d258b9f-85be-47cc-a145-93a15863a0ee" (UID: "2d258b9f-85be-47cc-a145-93a15863a0ee"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:53:11.862745 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.862722 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d258b9f-85be-47cc-a145-93a15863a0ee-kube-api-access-zxmq6" (OuterVolumeSpecName: "kube-api-access-zxmq6") pod "2d258b9f-85be-47cc-a145-93a15863a0ee" (UID: "2d258b9f-85be-47cc-a145-93a15863a0ee"). InnerVolumeSpecName "kube-api-access-zxmq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:53:11.960899 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.960827 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxmq6\" (UniqueName: \"kubernetes.io/projected/2d258b9f-85be-47cc-a145-93a15863a0ee-kube-api-access-zxmq6\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:53:11.960899 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.960857 2529 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-offline-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:53:11.960899 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.960873 2529 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d258b9f-85be-47cc-a145-93a15863a0ee-feast-data\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:53:11.960899 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.960882 2529 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-ui-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:53:11.960899 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.960890 2529 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-registry-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:53:11.960899 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.960897 2529 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d258b9f-85be-47cc-a145-93a15863a0ee-online-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:53:11.994897 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.994866 2529 generic.go:356] "Generic (PLEG): container finished" podID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerID="99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd" exitCode=137 Jul 28 05:53:11.994897 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.994891 2529 generic.go:356] "Generic (PLEG): container finished" podID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerID="5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0" exitCode=137 Jul 28 05:53:11.995070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.994911 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerDied","Data":"99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd"} Jul 28 05:53:11.995070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.994922 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" Jul 28 05:53:11.995070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.994952 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerDied","Data":"5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0"} Jul 28 05:53:11.995070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.994966 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j" event={"ID":"2d258b9f-85be-47cc-a145-93a15863a0ee","Type":"ContainerDied","Data":"8e881df972f352b931913218656d52528629cd0389c4d528ac8e2d42e1ad680a"} Jul 28 05:53:11.995070 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:11.994985 2529 scope.go:117] "RemoveContainer" containerID="a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda" Jul 28 05:53:12.004395 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.004379 2529 scope.go:117] "RemoveContainer" containerID="99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd" Jul 28 05:53:12.011959 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.011940 2529 scope.go:117] "RemoveContainer" containerID="1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f" Jul 28 05:53:12.016792 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.016770 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j"] Jul 28 05:53:12.019859 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.019842 2529 scope.go:117] "RemoveContainer" containerID="5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0" Jul 28 05:53:12.021685 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.021661 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2nb7j"] Jul 28 05:53:12.027275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.027250 2529 scope.go:117] "RemoveContainer" containerID="1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1" Jul 28 05:53:12.034598 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.034575 2529 scope.go:117] "RemoveContainer" containerID="64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb" Jul 28 05:53:12.041743 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.041725 2529 scope.go:117] "RemoveContainer" containerID="a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda" Jul 28 05:53:12.042015 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:53:12.041995 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda\": container with ID starting with a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda not found: ID does not exist" containerID="a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda" Jul 28 05:53:12.042063 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.042019 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda"} err="failed to get container status \"a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda\": rpc error: code = NotFound desc = could not find container \"a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda\": container with ID starting with a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda not found: ID does not exist" Jul 28 05:53:12.042063 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.042038 2529 scope.go:117] "RemoveContainer" containerID="99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd" Jul 28 05:53:12.042264 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:53:12.042246 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd\": container with ID starting with 99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd not found: ID does not exist" containerID="99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd" Jul 28 05:53:12.042298 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.042272 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd"} err="failed to get container status \"99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd\": rpc error: code = NotFound desc = could not find container \"99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd\": container with ID starting with 99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd not found: ID does not exist" Jul 28 05:53:12.042298 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.042289 2529 scope.go:117] "RemoveContainer" containerID="1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f" Jul 28 05:53:12.042524 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:53:12.042496 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f\": container with ID starting with 1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f not found: ID does not exist" containerID="1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f" Jul 28 05:53:12.042575 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.042533 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f"} err="failed to get container status \"1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f\": rpc error: code = NotFound desc = could not find container \"1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f\": container with ID starting with 1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f not found: ID does not exist" Jul 28 05:53:12.042575 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.042549 2529 scope.go:117] "RemoveContainer" containerID="5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0" Jul 28 05:53:12.042769 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:53:12.042755 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0\": container with ID starting with 5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0 not found: ID does not exist" containerID="5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0" Jul 28 05:53:12.042810 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.042772 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0"} err="failed to get container status \"5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0\": rpc error: code = NotFound desc = could not find container \"5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0\": container with ID starting with 5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0 not found: ID does not exist" Jul 28 05:53:12.042810 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.042787 2529 scope.go:117] "RemoveContainer" containerID="1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1" Jul 28 05:53:12.043000 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:53:12.042983 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1\": container with ID starting with 1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1 not found: ID does not exist" containerID="1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1" Jul 28 05:53:12.043034 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043006 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1"} err="failed to get container status \"1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1\": rpc error: code = NotFound desc = could not find container \"1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1\": container with ID starting with 1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1 not found: ID does not exist" Jul 28 05:53:12.043034 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043029 2529 scope.go:117] "RemoveContainer" containerID="64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb" Jul 28 05:53:12.043229 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:53:12.043214 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb\": container with ID starting with 64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb not found: ID does not exist" containerID="64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb" Jul 28 05:53:12.043272 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043232 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb"} err="failed to get container status \"64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb\": rpc error: code = NotFound desc = could not find container \"64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb\": container with ID starting with 64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb not found: ID does not exist" Jul 28 05:53:12.043272 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043244 2529 scope.go:117] "RemoveContainer" containerID="a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda" Jul 28 05:53:12.043436 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043422 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda"} err="failed to get container status \"a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda\": rpc error: code = NotFound desc = could not find container \"a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda\": container with ID starting with a2614787cccf1f13bb0e4e6bd6e182f4dd0c955bbd01cc0dca7a3751d6d41cda not found: ID does not exist" Jul 28 05:53:12.043474 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043435 2529 scope.go:117] "RemoveContainer" containerID="99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd" Jul 28 05:53:12.043660 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043643 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd"} err="failed to get container status \"99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd\": rpc error: code = NotFound desc = could not find container \"99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd\": container with ID starting with 99e6f1c214ec15a8bde55854a8df0fe8e88a0c5eb41e8a2b948f75d7da182bdd not found: ID does not exist" Jul 28 05:53:12.043705 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043661 2529 scope.go:117] "RemoveContainer" containerID="1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f" Jul 28 05:53:12.043890 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043874 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f"} err="failed to get container status \"1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f\": rpc error: code = NotFound desc = could not find container \"1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f\": container with ID starting with 1319c6fd2c899f20793a2dfa8804f8963bdaaed02454ccbc8d7fa479e279d63f not found: ID does not exist" Jul 28 05:53:12.043948 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.043891 2529 scope.go:117] "RemoveContainer" containerID="5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0" Jul 28 05:53:12.044130 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.044113 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0"} err="failed to get container status \"5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0\": rpc error: code = NotFound desc = could not find container \"5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0\": container with ID starting with 5c2ee08cb2033c6b1a8d299dff6a5c0c96daadbd5686dd8090441e5cd8c44ff0 not found: ID does not exist" Jul 28 05:53:12.044170 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.044131 2529 scope.go:117] "RemoveContainer" containerID="1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1" Jul 28 05:53:12.044355 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.044337 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1"} err="failed to get container status \"1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1\": rpc error: code = NotFound desc = could not find container \"1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1\": container with ID starting with 1b22ec3dacf4264a5ce5ab933489d14739c6ca63e6c282e26231bad25339c8c1 not found: ID does not exist" Jul 28 05:53:12.044428 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.044357 2529 scope.go:117] "RemoveContainer" containerID="64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb" Jul 28 05:53:12.044622 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:12.044601 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb"} err="failed to get container status \"64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb\": rpc error: code = NotFound desc = could not find container \"64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb\": container with ID starting with 64a58da6d721262d1c3322658332206afdaa4756645d72ae841b458d8d258bfb not found: ID does not exist" Jul 28 05:53:13.585640 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:53:13.585609 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" path="/var/lib/kubelet/pods/2d258b9f-85be-47cc-a145-93a15863a0ee/volumes" Jul 28 05:54:04.890043 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.889966 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq"] Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890314 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="feast-apply" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890326 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="feast-apply" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890343 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="online" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890349 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="online" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890356 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="offline" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890365 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="offline" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890378 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="feast-init" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890384 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="feast-init" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890392 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="registry" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890397 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="registry" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890405 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="ui" Jul 28 05:54:04.890469 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890410 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="ui" Jul 28 05:54:04.891088 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890484 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="offline" Jul 28 05:54:04.891088 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890496 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="ui" Jul 28 05:54:04.891088 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890511 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="registry" Jul 28 05:54:04.891088 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.890534 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d258b9f-85be-47cc-a145-93a15863a0ee" containerName="online" Jul 28 05:54:04.894087 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.894071 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:04.896396 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.896373 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 28 05:54:04.896538 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.896465 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 28 05:54:04.896538 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.896530 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 28 05:54:04.896651 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.896612 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-z2jv4\"" Jul 28 05:54:04.896745 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.896731 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 28 05:54:04.896827 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.896809 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 28 05:54:04.896867 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.896854 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 28 05:54:04.904428 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.904407 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq"] Jul 28 05:54:04.913179 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.913154 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:04.913275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.913184 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:04.913275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.913199 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:04.913275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.913217 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:04.913275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.913251 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:04.913412 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:04.913297 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bcfn\" (UniqueName: \"kubernetes.io/projected/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-kube-api-access-4bcfn\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.013919 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.013891 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.013919 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.013923 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.014102 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.013943 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.014102 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.013971 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.014102 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.014002 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.014102 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.014046 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bcfn\" (UniqueName: \"kubernetes.io/projected/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-kube-api-access-4bcfn\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.014102 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:54:05.014058 2529 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 28 05:54:05.014336 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:54:05.014058 2529 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 05:54:05.014336 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:54:05.014133 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls podName:92a85ca1-b7ed-49e5-acc0-0ccc59d11b59 nodeName:}" failed. No retries permitted until 2026-07-28 05:54:05.514113903 +0000 UTC m=+1908.446149044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls") pod "feast-simple-feast-remote-setup-db6955c6c-l9vcq" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 28 05:54:05.014336 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:54:05.014063 2529 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 05:54:05.014336 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:54:05.014204 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-online-tls podName:92a85ca1-b7ed-49e5-acc0-0ccc59d11b59 nodeName:}" failed. No retries permitted until 2026-07-28 05:54:05.514185011 +0000 UTC m=+1908.446220165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-online-tls") pod "feast-simple-feast-remote-setup-db6955c6c-l9vcq" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59") : secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 05:54:05.014336 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:54:05.014250 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-ui-tls podName:92a85ca1-b7ed-49e5-acc0-0ccc59d11b59 nodeName:}" failed. No retries permitted until 2026-07-28 05:54:05.51423602 +0000 UTC m=+1908.446271176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-ui-tls") pod "feast-simple-feast-remote-setup-db6955c6c-l9vcq" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 05:54:05.014336 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.014299 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.014649 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.014632 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.023228 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.023205 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bcfn\" (UniqueName: \"kubernetes.io/projected/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-kube-api-access-4bcfn\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.520299 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.520267 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.520299 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.520302 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.520610 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.520321 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.520610 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:54:05.520428 2529 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 28 05:54:05.520610 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:54:05.520505 2529 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls podName:92a85ca1-b7ed-49e5-acc0-0ccc59d11b59 nodeName:}" failed. No retries permitted until 2026-07-28 05:54:06.520486998 +0000 UTC m=+1909.452522136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls") pod "feast-simple-feast-remote-setup-db6955c6c-l9vcq" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 28 05:54:05.522681 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.522658 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:05.522790 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:05.522702 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:06.527329 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:06.527296 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:06.529608 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:06.529589 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-l9vcq\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:06.703414 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:06.703382 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:06.844340 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:06.844275 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq"] Jul 28 05:54:06.846639 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:54:06.846611 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92a85ca1_b7ed_49e5_acc0_0ccc59d11b59.slice/crio-9addf0581e765146d62d1104b47a7013a31328e6a959e651af7e816fc387cc72 WatchSource:0}: Error finding container 9addf0581e765146d62d1104b47a7013a31328e6a959e651af7e816fc387cc72: Status 404 returned error can't find the container with id 9addf0581e765146d62d1104b47a7013a31328e6a959e651af7e816fc387cc72 Jul 28 05:54:07.228227 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:07.228185 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerStarted","Data":"ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96"} Jul 28 05:54:07.228387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:07.228236 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerStarted","Data":"9addf0581e765146d62d1104b47a7013a31328e6a959e651af7e816fc387cc72"} Jul 28 05:54:12.251718 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:12.251688 2529 generic.go:356] "Generic (PLEG): container finished" podID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerID="ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96" exitCode=0 Jul 28 05:54:12.252113 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:12.251728 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerDied","Data":"ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96"} Jul 28 05:54:13.257478 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:13.257440 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerStarted","Data":"a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf"} Jul 28 05:54:18.281871 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:18.281832 2529 generic.go:356] "Generic (PLEG): container finished" podID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerID="a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf" exitCode=0 Jul 28 05:54:18.282359 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:18.281908 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerDied","Data":"a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf"} Jul 28 05:54:19.291202 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:19.291157 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerStarted","Data":"673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1"} Jul 28 05:54:19.291695 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:19.291219 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerStarted","Data":"c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1"} Jul 28 05:54:19.291695 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:19.291261 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerStarted","Data":"32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9"} Jul 28 05:54:19.314754 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:19.314685 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podStartSLOduration=15.314661423 podStartE2EDuration="15.314661423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:54:19.312707712 +0000 UTC m=+1922.244742872" watchObservedRunningTime="2026-07-28 05:54:19.314661423 +0000 UTC m=+1922.246696587" Jul 28 05:54:21.704477 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:21.704437 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:21.705088 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:21.705060 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:21.705088 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:21.705088 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:21.706612 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:21.706571 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" Jul 28 05:54:21.706725 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:21.706591 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Jul 28 05:54:21.706792 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:21.706630 2529 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jul 28 05:54:24.705194 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:24.705163 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:24.705665 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:24.705348 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:24.705748 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:24.705701 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:24.705748 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:24.705733 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:24.705989 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:24.705971 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:24.706167 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:24.706155 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:24.711561 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:24.711540 2529 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:25.331859 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:25.331826 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:54:25.336300 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:54:25.336276 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:55:27.936505 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:27.936419 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq"] Jul 28 05:55:27.937073 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:27.936726 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="online" containerID="cri-o://32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9" gracePeriod=30 Jul 28 05:55:27.937073 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:27.936791 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="offline" containerID="cri-o://c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1" gracePeriod=30 Jul 28 05:55:27.937073 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:27.936793 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="ui" containerID="cri-o://673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1" gracePeriod=30 Jul 28 05:55:28.106810 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:28.106768 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv"] Jul 28 05:55:28.107101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:28.107049 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="registry" containerID="cri-o://224144a347c4f3cb8b2c4fd364c2ef93a37a53d1fc7616becad381be96097d8b" gracePeriod=30 Jul 28 05:55:28.107219 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:28.107153 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="ui" containerID="cri-o://d7beb44f2b3a237f901c49811c60c303b38612c4aa68ac52cb6817ac592538bc" gracePeriod=30 Jul 28 05:55:28.107291 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:28.107231 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="online" containerID="cri-o://8e07a5e673032c5398104b046803c53d30cce188fa771841c1dd99bdc54c7b0f" gracePeriod=30 Jul 28 05:55:28.107360 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:28.107321 2529 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="offline" containerID="cri-o://a443f1258f3bee295f6c458714f1769f23f186600e62ff2aa6471947f62a850e" gracePeriod=30 Jul 28 05:55:28.602356 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:28.602324 2529 generic.go:356] "Generic (PLEG): container finished" podID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerID="673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1" exitCode=0 Jul 28 05:55:28.602509 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:28.602403 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerDied","Data":"673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1"} Jul 28 05:55:29.609306 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:29.609268 2529 generic.go:356] "Generic (PLEG): container finished" podID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerID="32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9" exitCode=0 Jul 28 05:55:29.609763 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:29.609337 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerDied","Data":"32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9"} Jul 28 05:55:29.612036 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:29.612015 2529 generic.go:356] "Generic (PLEG): container finished" podID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerID="d7beb44f2b3a237f901c49811c60c303b38612c4aa68ac52cb6817ac592538bc" exitCode=0 Jul 28 05:55:29.612036 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:29.612033 2529 generic.go:356] "Generic (PLEG): container finished" podID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerID="8e07a5e673032c5398104b046803c53d30cce188fa771841c1dd99bdc54c7b0f" exitCode=0 Jul 28 05:55:29.612165 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:29.612081 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerDied","Data":"d7beb44f2b3a237f901c49811c60c303b38612c4aa68ac52cb6817ac592538bc"} Jul 28 05:55:29.612165 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:29.612111 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerDied","Data":"8e07a5e673032c5398104b046803c53d30cce188fa771841c1dd99bdc54c7b0f"} Jul 28 05:55:31.950868 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:31.950819 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" Jul 28 05:55:31.951306 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:31.950916 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Jul 28 05:55:34.706683 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:34.706642 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Jul 28 05:55:35.332275 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:35.332227 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jul 28 05:55:41.951212 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:41.951162 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" Jul 28 05:55:41.951614 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:41.951175 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Jul 28 05:55:44.706242 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:44.706197 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Jul 28 05:55:45.332587 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:45.332546 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jul 28 05:55:51.950599 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:51.950554 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" Jul 28 05:55:51.950599 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:51.950593 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Jul 28 05:55:51.951010 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:51.950686 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:55:51.951010 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:51.950752 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:55:54.706109 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:54.706059 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Jul 28 05:55:54.706571 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:54.706176 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:55:55.332609 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:55.332569 2529 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jul 28 05:55:55.332792 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:55.332681 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:55:58.587659 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.587632 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:55:58.732433 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.732342 2529 generic.go:356] "Generic (PLEG): container finished" podID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerID="c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1" exitCode=137 Jul 28 05:55:58.732433 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.732421 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" Jul 28 05:55:58.732654 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.732423 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerDied","Data":"c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1"} Jul 28 05:55:58.732654 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.732478 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq" event={"ID":"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59","Type":"ContainerDied","Data":"9addf0581e765146d62d1104b47a7013a31328e6a959e651af7e816fc387cc72"} Jul 28 05:55:58.732654 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.732504 2529 scope.go:117] "RemoveContainer" containerID="673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1" Jul 28 05:55:58.734318 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.734294 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-ui-tls\") pod \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " Jul 28 05:55:58.734436 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.734330 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-registry-tls\") pod \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " Jul 28 05:55:58.734436 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.734355 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bcfn\" (UniqueName: \"kubernetes.io/projected/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-kube-api-access-4bcfn\") pod \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " Jul 28 05:55:58.734436 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.734384 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls\") pod \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " Jul 28 05:55:58.734436 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.734432 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-feast-data\") pod \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " Jul 28 05:55:58.734709 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.734488 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-online-tls\") pod \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\" (UID: \"92a85ca1-b7ed-49e5-acc0-0ccc59d11b59\") " Jul 28 05:55:58.734709 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.734692 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:55:58.734886 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.734862 2529 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-registry-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:58.735283 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.735259 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-feast-data" (OuterVolumeSpecName: "feast-data") pod "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:55:58.736385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.736358 2529 generic.go:356] "Generic (PLEG): container finished" podID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerID="a443f1258f3bee295f6c458714f1769f23f186600e62ff2aa6471947f62a850e" exitCode=137 Jul 28 05:55:58.736385 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.736382 2529 generic.go:356] "Generic (PLEG): container finished" podID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerID="224144a347c4f3cb8b2c4fd364c2ef93a37a53d1fc7616becad381be96097d8b" exitCode=137 Jul 28 05:55:58.736803 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.736466 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerDied","Data":"a443f1258f3bee295f6c458714f1769f23f186600e62ff2aa6471947f62a850e"} Jul 28 05:55:58.736803 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.736498 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerDied","Data":"224144a347c4f3cb8b2c4fd364c2ef93a37a53d1fc7616becad381be96097d8b"} Jul 28 05:55:58.737680 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.737653 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-kube-api-access-4bcfn" (OuterVolumeSpecName: "kube-api-access-4bcfn") pod "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59"). InnerVolumeSpecName "kube-api-access-4bcfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:55:58.737680 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.737659 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:55:58.737827 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.737742 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:55:58.739828 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.739803 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-online-tls" (OuterVolumeSpecName: "online-tls") pod "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" (UID: "92a85ca1-b7ed-49e5-acc0-0ccc59d11b59"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:55:58.742209 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.742189 2529 scope.go:117] "RemoveContainer" containerID="c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1" Jul 28 05:55:58.750025 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.750004 2529 scope.go:117] "RemoveContainer" containerID="32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9" Jul 28 05:55:58.758092 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.758074 2529 scope.go:117] "RemoveContainer" containerID="a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf" Jul 28 05:55:58.766050 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.766034 2529 scope.go:117] "RemoveContainer" containerID="ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96" Jul 28 05:55:58.768404 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.768384 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:55:58.773202 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.773185 2529 scope.go:117] "RemoveContainer" containerID="673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1" Jul 28 05:55:58.773440 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:55:58.773423 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1\": container with ID starting with 673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1 not found: ID does not exist" containerID="673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1" Jul 28 05:55:58.773486 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.773449 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1"} err="failed to get container status \"673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1\": rpc error: code = NotFound desc = could not find container \"673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1\": container with ID starting with 673278b104807e22b80589a21ba79e289643e6f1c52c142740ad6716e95293c1 not found: ID does not exist" Jul 28 05:55:58.773486 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.773465 2529 scope.go:117] "RemoveContainer" containerID="c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1" Jul 28 05:55:58.773694 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:55:58.773678 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1\": container with ID starting with c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1 not found: ID does not exist" containerID="c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1" Jul 28 05:55:58.773739 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.773698 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1"} err="failed to get container status \"c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1\": rpc error: code = NotFound desc = could not find container \"c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1\": container with ID starting with c7fbfcf1e68c44ef3934cdd102f3d116ed3adf4f2cb51ae0dc9ed970fe3031e1 not found: ID does not exist" Jul 28 05:55:58.773739 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.773711 2529 scope.go:117] "RemoveContainer" containerID="32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9" Jul 28 05:55:58.773943 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:55:58.773921 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9\": container with ID starting with 32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9 not found: ID does not exist" containerID="32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9" Jul 28 05:55:58.774015 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.773951 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9"} err="failed to get container status \"32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9\": rpc error: code = NotFound desc = could not find container \"32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9\": container with ID starting with 32f732e656ce061cd041f92555c1fdc2079c8c51716575805e71dcc73939d2a9 not found: ID does not exist" Jul 28 05:55:58.774015 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.773973 2529 scope.go:117] "RemoveContainer" containerID="a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf" Jul 28 05:55:58.774193 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:55:58.774173 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf\": container with ID starting with a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf not found: ID does not exist" containerID="a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf" Jul 28 05:55:58.774232 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.774199 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf"} err="failed to get container status \"a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf\": rpc error: code = NotFound desc = could not find container \"a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf\": container with ID starting with a1cc5c306c7259d863b7c4f394070bd84d3573f76ebf324367b7914e63a273bf not found: ID does not exist" Jul 28 05:55:58.774232 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.774214 2529 scope.go:117] "RemoveContainer" containerID="ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96" Jul 28 05:55:58.774395 ip-10-0-137-238 kubenswrapper[2529]: E0728 05:55:58.774379 2529 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96\": container with ID starting with ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96 not found: ID does not exist" containerID="ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96" Jul 28 05:55:58.774442 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.774400 2529 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96"} err="failed to get container status \"ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96\": rpc error: code = NotFound desc = could not find container \"ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96\": container with ID starting with ba1a612878ebad29444ca961096969530f9b759e33a02fc866adac88e6742e96 not found: ID does not exist" Jul 28 05:55:58.836305 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.836281 2529 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-ui-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:58.836305 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.836302 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bcfn\" (UniqueName: \"kubernetes.io/projected/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-kube-api-access-4bcfn\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:58.836460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.836312 2529 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-offline-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:58.836460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.836324 2529 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-feast-data\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:58.836460 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.836331 2529 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59-online-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:58.937315 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.937280 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-online-tls\") pod \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " Jul 28 05:55:58.937681 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.937340 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-feast-data\") pod \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " Jul 28 05:55:58.937681 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.937390 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls\") pod \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " Jul 28 05:55:58.937681 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.937478 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-registry-tls\") pod \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " Jul 28 05:55:58.937681 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.937545 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-ui-tls\") pod \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " Jul 28 05:55:58.937681 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.937582 2529 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9qfw\" (UniqueName: \"kubernetes.io/projected/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-kube-api-access-v9qfw\") pod \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\" (UID: \"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac\") " Jul 28 05:55:58.938041 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.938007 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-feast-data" (OuterVolumeSpecName: "feast-data") pod "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" (UID: "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:55:58.939648 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.939622 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" (UID: "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:55:58.939765 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.939648 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" (UID: "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:55:58.939765 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.939672 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-online-tls" (OuterVolumeSpecName: "online-tls") pod "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" (UID: "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:55:58.939883 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.939838 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" (UID: "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:55:58.939988 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:58.939969 2529 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-kube-api-access-v9qfw" (OuterVolumeSpecName: "kube-api-access-v9qfw") pod "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" (UID: "e2a6c7d9-341d-4ff5-8997-27993fa0d2ac"). InnerVolumeSpecName "kube-api-access-v9qfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:55:59.038328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.038252 2529 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-registry-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:59.038328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.038275 2529 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-ui-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:59.038328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.038285 2529 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9qfw\" (UniqueName: \"kubernetes.io/projected/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-kube-api-access-v9qfw\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:59.038328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.038294 2529 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-online-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:59.038328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.038302 2529 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-feast-data\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:59.038328 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.038309 2529 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac-offline-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\"" Jul 28 05:55:59.053788 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.053761 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq"] Jul 28 05:55:59.057440 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.057416 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-l9vcq"] Jul 28 05:55:59.584999 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.584967 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" path="/var/lib/kubelet/pods/92a85ca1-b7ed-49e5-acc0-0ccc59d11b59/volumes" Jul 28 05:55:59.743396 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.743371 2529 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" Jul 28 05:55:59.743814 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.743376 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv" event={"ID":"e2a6c7d9-341d-4ff5-8997-27993fa0d2ac","Type":"ContainerDied","Data":"8665d9989afe0d502e3d13d1f8c2cad391159646c6c9f9780026e6f16d536c12"} Jul 28 05:55:59.743814 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.743491 2529 scope.go:117] "RemoveContainer" containerID="d7beb44f2b3a237f901c49811c60c303b38612c4aa68ac52cb6817ac592538bc" Jul 28 05:55:59.752235 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.752216 2529 scope.go:117] "RemoveContainer" containerID="a443f1258f3bee295f6c458714f1769f23f186600e62ff2aa6471947f62a850e" Jul 28 05:55:59.760750 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.760702 2529 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv"] Jul 28 05:55:59.760809 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.760766 2529 scope.go:117] "RemoveContainer" containerID="8e07a5e673032c5398104b046803c53d30cce188fa771841c1dd99bdc54c7b0f" Jul 28 05:55:59.765759 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.765736 2529 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-v9rbv"] Jul 28 05:55:59.768870 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.768855 2529 scope.go:117] "RemoveContainer" containerID="224144a347c4f3cb8b2c4fd364c2ef93a37a53d1fc7616becad381be96097d8b" Jul 28 05:55:59.776357 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.776341 2529 scope.go:117] "RemoveContainer" containerID="955ae72e69c083a0473a9b27b74acff83596eddabe52fcbe04b256d7f2ed8872" Jul 28 05:55:59.783631 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:55:59.783607 2529 scope.go:117] "RemoveContainer" containerID="58999fc65befec5a3f486d559382a7c2643749e30f7c0daf0a0730fb77ace6ef" Jul 28 05:56:01.585681 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:01.585651 2529 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" path="/var/lib/kubelet/pods/e2a6c7d9-341d-4ff5-8997-27993fa0d2ac/volumes" Jul 28 05:56:36.534276 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:36.534249 2529 ???:1] "http: TLS handshake error from 10.0.141.124:58248: EOF" Jul 28 05:56:36.540735 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:36.540712 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r52w4_7ce96221-80e8-4768-a905-818462349deb/global-pull-secret-syncer/0.log" Jul 28 05:56:36.656299 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:36.656274 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qhsdk_c3559246-3cfe-44e7-95ea-6a974178f96e/konnectivity-agent/0.log" Jul 28 05:56:36.707649 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:36.707624 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-238.ec2.internal_6d68380c15b9e00b00440ef022ddd164/haproxy/0.log" Jul 28 05:56:40.168530 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.168429 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd0cb4ca-b424-4224-87f5-6eff56c8ff83/alertmanager/0.log" Jul 28 05:56:40.191885 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.191855 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd0cb4ca-b424-4224-87f5-6eff56c8ff83/config-reloader/0.log" Jul 28 05:56:40.221670 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.221646 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd0cb4ca-b424-4224-87f5-6eff56c8ff83/kube-rbac-proxy-web/0.log" Jul 28 05:56:40.247042 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.247020 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd0cb4ca-b424-4224-87f5-6eff56c8ff83/kube-rbac-proxy/0.log" Jul 28 05:56:40.273059 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.273038 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd0cb4ca-b424-4224-87f5-6eff56c8ff83/kube-rbac-proxy-metric/0.log" Jul 28 05:56:40.295110 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.295088 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd0cb4ca-b424-4224-87f5-6eff56c8ff83/prom-label-proxy/0.log" Jul 28 05:56:40.314287 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.314266 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd0cb4ca-b424-4224-87f5-6eff56c8ff83/init-config-reloader/0.log" Jul 28 05:56:40.593052 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.593023 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j6dql_eaf66664-e8b3-47c8-8d0c-b3dac0d9962a/node-exporter/0.log" Jul 28 05:56:40.613024 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.613000 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j6dql_eaf66664-e8b3-47c8-8d0c-b3dac0d9962a/kube-rbac-proxy/0.log" Jul 28 05:56:40.638577 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.638558 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j6dql_eaf66664-e8b3-47c8-8d0c-b3dac0d9962a/init-textfile/0.log" Jul 28 05:56:40.800000 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.799974 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-szrmd_20f55225-ec6f-4896-9a6a-dc93a51601ff/kube-rbac-proxy-main/0.log" Jul 28 05:56:40.834740 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.834716 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-szrmd_20f55225-ec6f-4896-9a6a-dc93a51601ff/kube-rbac-proxy-self/0.log" Jul 28 05:56:40.878108 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.878030 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-szrmd_20f55225-ec6f-4896-9a6a-dc93a51601ff/openshift-state-metrics/0.log" Jul 28 05:56:40.931719 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.931697 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69ccc8ce-73fd-47c0-9788-05d38af066cd/prometheus/0.log" Jul 28 05:56:40.967898 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:40.967878 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69ccc8ce-73fd-47c0-9788-05d38af066cd/config-reloader/0.log" Jul 28 05:56:41.013946 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.013926 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69ccc8ce-73fd-47c0-9788-05d38af066cd/thanos-sidecar/0.log" Jul 28 05:56:41.049445 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.049422 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69ccc8ce-73fd-47c0-9788-05d38af066cd/kube-rbac-proxy-web/0.log" Jul 28 05:56:41.088780 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.088753 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69ccc8ce-73fd-47c0-9788-05d38af066cd/kube-rbac-proxy/0.log" Jul 28 05:56:41.130297 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.130232 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69ccc8ce-73fd-47c0-9788-05d38af066cd/kube-rbac-proxy-thanos/0.log" Jul 28 05:56:41.174211 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.174190 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69ccc8ce-73fd-47c0-9788-05d38af066cd/init-config-reloader/0.log" Jul 28 05:56:41.232629 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.232605 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-95z4b_92cb6b30-f216-461a-97da-1766f27923d6/prometheus-operator/0.log" Jul 28 05:56:41.274753 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.274722 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-95z4b_92cb6b30-f216-461a-97da-1766f27923d6/kube-rbac-proxy/0.log" Jul 28 05:56:41.325697 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.325677 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-jq5lw_ac096c81-397b-446c-83b6-823b10249e8b/prometheus-operator-admission-webhook/0.log" Jul 28 05:56:41.547105 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.547081 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf4cb454b-kvwft_77bbca72-d81d-494e-a5c0-b1119403e412/thanos-query/0.log" Jul 28 05:56:41.576278 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.576241 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf4cb454b-kvwft_77bbca72-d81d-494e-a5c0-b1119403e412/kube-rbac-proxy-web/0.log" Jul 28 05:56:41.609398 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.609380 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf4cb454b-kvwft_77bbca72-d81d-494e-a5c0-b1119403e412/kube-rbac-proxy/0.log" Jul 28 05:56:41.629454 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.629429 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf4cb454b-kvwft_77bbca72-d81d-494e-a5c0-b1119403e412/prom-label-proxy/0.log" Jul 28 05:56:41.653671 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.653654 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf4cb454b-kvwft_77bbca72-d81d-494e-a5c0-b1119403e412/kube-rbac-proxy-rules/0.log" Jul 28 05:56:41.673785 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:41.673762 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf4cb454b-kvwft_77bbca72-d81d-494e-a5c0-b1119403e412/kube-rbac-proxy-metrics/0.log" Jul 28 05:56:43.066940 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.066860 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/2.log" Jul 28 05:56:43.074144 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.074115 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8jw4s_7f5fa4b6-32bf-4260-a75f-c28d53326d84/console-operator/3.log" Jul 28 05:56:43.475829 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.475800 2529 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n"] Jul 28 05:56:43.476119 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476107 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="feast-apply" Jul 28 05:56:43.476169 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476120 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="feast-apply" Jul 28 05:56:43.476169 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476132 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="ui" Jul 28 05:56:43.476169 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476138 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="ui" Jul 28 05:56:43.476169 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476147 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="online" Jul 28 05:56:43.476169 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476152 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="online" Jul 28 05:56:43.476169 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476159 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="feast-init" Jul 28 05:56:43.476169 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476165 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="feast-init" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476177 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="registry" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476182 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="registry" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476194 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="offline" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476200 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="offline" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476208 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="ui" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476213 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="ui" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476221 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="online" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476226 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="online" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476234 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="feast-apply" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476239 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="feast-apply" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476246 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="offline" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476251 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="offline" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476257 2529 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="feast-init" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476262 2529 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="feast-init" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476307 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="offline" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476316 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="online" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476322 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a85ca1-b7ed-49e5-acc0-0ccc59d11b59" containerName="ui" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476329 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="ui" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476335 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="registry" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476340 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="offline" Jul 28 05:56:43.476387 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.476345 2529 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a6c7d9-341d-4ff5-8997-27993fa0d2ac" containerName="online" Jul 28 05:56:43.479450 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.479434 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.481723 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.481701 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26j78\"/\"openshift-service-ca.crt\"" Jul 28 05:56:43.482557 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.482539 2529 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26j78\"/\"kube-root-ca.crt\"" Jul 28 05:56:43.482661 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.482578 2529 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26j78\"/\"default-dockercfg-8bcm9\"" Jul 28 05:56:43.486965 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.486942 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n"] Jul 28 05:56:43.616682 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.616647 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-lib-modules\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.616858 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.616688 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-proc\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.616858 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.616764 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-podres\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.616858 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.616839 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-sys\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.616994 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.616890 2529 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbnr\" (UniqueName: \"kubernetes.io/projected/49c3ed75-4ec4-4825-af56-88e20c6a1c05-kube-api-access-zqbnr\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.717994 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.717949 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-podres\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.717994 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.718002 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-sys\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.718199 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.718051 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbnr\" (UniqueName: \"kubernetes.io/projected/49c3ed75-4ec4-4825-af56-88e20c6a1c05-kube-api-access-zqbnr\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.718199 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.718102 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-lib-modules\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.718199 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.718123 2529 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-proc\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.718199 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.718197 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-proc\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.718474 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.718444 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-sys\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.718474 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.718467 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-podres\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.718702 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.718495 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49c3ed75-4ec4-4825-af56-88e20c6a1c05-lib-modules\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.725281 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.725264 2529 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbnr\" (UniqueName: \"kubernetes.io/projected/49c3ed75-4ec4-4825-af56-88e20c6a1c05-kube-api-access-zqbnr\") pod \"perf-node-gather-daemonset-zns7n\" (UID: \"49c3ed75-4ec4-4825-af56-88e20c6a1c05\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.790446 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.790350 2529 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:43.914577 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.914553 2529 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n"] Jul 28 05:56:43.916869 ip-10-0-137-238 kubenswrapper[2529]: W0728 05:56:43.916843 2529 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49c3ed75_4ec4_4825_af56_88e20c6a1c05.slice/crio-71aaf6a611c210051f80a080a9a127731b94a9fcd10239cef008f46760cb0d3f WatchSource:0}: Error finding container 71aaf6a611c210051f80a080a9a127731b94a9fcd10239cef008f46760cb0d3f: Status 404 returned error can't find the container with id 71aaf6a611c210051f80a080a9a127731b94a9fcd10239cef008f46760cb0d3f Jul 28 05:56:43.944205 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:43.944170 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-7kh86_bbb7476a-298d-402f-b619-96146cf2be77/volume-data-source-validator/0.log" Jul 28 05:56:44.714325 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:44.714298 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n7tqp_c0855a53-29d0-4bad-95a4-e0696e3357c4/dns/0.log" Jul 28 05:56:44.734987 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:44.734965 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n7tqp_c0855a53-29d0-4bad-95a4-e0696e3357c4/kube-rbac-proxy/0.log" Jul 28 05:56:44.756182 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:44.756164 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6cnbv_6e82bb4e-d528-49e5-8c0d-02eb57c2e399/dns-node-resolver/0.log" Jul 28 05:56:44.925116 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:44.925084 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" event={"ID":"49c3ed75-4ec4-4825-af56-88e20c6a1c05","Type":"ContainerStarted","Data":"c72a3ebb7d1a07997c14428297a9200838d63b3f81399208d330122d1fdb4bbc"} Jul 28 05:56:44.925282 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:44.925159 2529 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" event={"ID":"49c3ed75-4ec4-4825-af56-88e20c6a1c05","Type":"ContainerStarted","Data":"71aaf6a611c210051f80a080a9a127731b94a9fcd10239cef008f46760cb0d3f"} Jul 28 05:56:44.925282 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:44.925220 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:44.939347 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:44.939302 2529 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" podStartSLOduration=1.939289869 podStartE2EDuration="1.939289869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:56:44.938241854 +0000 UTC m=+2067.870277017" watchObservedRunningTime="2026-07-28 05:56:44.939289869 +0000 UTC m=+2067.871325028" Jul 28 05:56:45.152904 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:45.152876 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85c655d49c-nqhld_f14685e5-9535-4ab2-80fc-063bbe29acdb/registry/0.log" Jul 28 05:56:45.214789 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:45.214760 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6ff9_9c86d6b2-647e-4409-9fb3-ca3861e7b9d4/node-ca/0.log" Jul 28 05:56:46.082183 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:46.082150 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87bqg_f4600925-15b9-4d3e-8329-dfa4d59fe6f3/serve-healthcheck-canary/0.log" Jul 28 05:56:46.437643 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:46.437612 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9tcqp_2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4/kube-rbac-proxy/0.log" Jul 28 05:56:46.457307 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:46.457282 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9tcqp_2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4/exporter/0.log" Jul 28 05:56:46.477152 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:46.477130 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9tcqp_2a7d2c11-7a94-45a0-b3ef-bc6b2915bcb4/extractor/0.log" Jul 28 05:56:50.667277 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:50.667238 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr_fbe9002f-266b-4831-9c2c-80d889c2eeab/kube-storage-version-migrator-operator/1.log" Jul 28 05:56:50.668832 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:50.668807 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-zhvkr_fbe9002f-266b-4831-9c2c-80d889c2eeab/kube-storage-version-migrator-operator/0.log" Jul 28 05:56:50.939561 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:50.939472 2529 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-zns7n" Jul 28 05:56:51.700443 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:51.700418 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxvrw_543e0b9a-76d9-4728-8cdb-3ae735ec3dd7/kube-multus-additional-cni-plugins/0.log" Jul 28 05:56:51.724319 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:51.724293 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxvrw_543e0b9a-76d9-4728-8cdb-3ae735ec3dd7/egress-router-binary-copy/0.log" Jul 28 05:56:51.743350 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:51.743327 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxvrw_543e0b9a-76d9-4728-8cdb-3ae735ec3dd7/cni-plugins/0.log" Jul 28 05:56:51.761458 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:51.761440 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxvrw_543e0b9a-76d9-4728-8cdb-3ae735ec3dd7/bond-cni-plugin/0.log" Jul 28 05:56:51.779974 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:51.779954 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxvrw_543e0b9a-76d9-4728-8cdb-3ae735ec3dd7/routeoverride-cni/0.log" Jul 28 05:56:51.798796 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:51.798773 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxvrw_543e0b9a-76d9-4728-8cdb-3ae735ec3dd7/whereabouts-cni-bincopy/0.log" Jul 28 05:56:51.817816 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:51.817787 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xxvrw_543e0b9a-76d9-4728-8cdb-3ae735ec3dd7/whereabouts-cni/0.log" Jul 28 05:56:51.852505 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:51.852484 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ctmks_72c5583a-0e4e-42b6-b0d0-da18faa742ff/kube-multus/0.log" Jul 28 05:56:52.023101 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:52.023021 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qntsm_a585f5cd-60e1-4a50-acdc-2e508fcbbdca/network-metrics-daemon/0.log" Jul 28 05:56:52.043197 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:52.043146 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qntsm_a585f5cd-60e1-4a50-acdc-2e508fcbbdca/kube-rbac-proxy/0.log" Jul 28 05:56:53.055035 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:53.054951 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmnb_e0cf60e9-379a-4455-b9c5-6c72caa1774e/ovn-controller/0.log" Jul 28 05:56:53.069210 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:53.069176 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmnb_e0cf60e9-379a-4455-b9c5-6c72caa1774e/ovn-acl-logging/0.log" Jul 28 05:56:53.095375 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:53.095353 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmnb_e0cf60e9-379a-4455-b9c5-6c72caa1774e/kube-rbac-proxy-node/0.log" Jul 28 05:56:53.117742 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:53.117721 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmnb_e0cf60e9-379a-4455-b9c5-6c72caa1774e/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 05:56:53.132944 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:53.132920 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmnb_e0cf60e9-379a-4455-b9c5-6c72caa1774e/northd/0.log" Jul 28 05:56:53.152140 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:53.152120 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmnb_e0cf60e9-379a-4455-b9c5-6c72caa1774e/nbdb/0.log" Jul 28 05:56:53.172012 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:53.171995 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmnb_e0cf60e9-379a-4455-b9c5-6c72caa1774e/sbdb/0.log" Jul 28 05:56:53.228703 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:53.228673 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpmnb_e0cf60e9-379a-4455-b9c5-6c72caa1774e/ovnkube-controller/0.log" Jul 28 05:56:54.339550 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:54.339504 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-srm2m_f678dcbc-bc33-443a-b1bb-e119d0818a10/network-check-target-container/0.log" Jul 28 05:56:55.109045 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:55.109015 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-7rc59_262fafeb-4ba5-4b6e-a870-92f41b198b15/iptables-alerter/0.log" Jul 28 05:56:55.670565 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:55.670539 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-4wzrz_681f5ec2-4771-4e6a-b539-8b88971c205e/tuned/0.log" Jul 28 05:56:58.019011 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:58.018980 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-5d6c75f84f-ff9vx_92772616-27ea-4ff3-a422-a9c09ddbc1c0/service-ca-operator/1.log" Jul 28 05:56:58.020616 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:58.020589 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-5d6c75f84f-ff9vx_92772616-27ea-4ff3-a422-a9c09ddbc1c0/service-ca-operator/0.log" Jul 28 05:56:58.719095 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:58.719065 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-qvq6l_c4033dc0-62e3-4974-9d66-c98ea5033f7f/csi-driver/0.log" Jul 28 05:56:58.741889 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:58.741864 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-qvq6l_c4033dc0-62e3-4974-9d66-c98ea5033f7f/csi-node-driver-registrar/0.log" Jul 28 05:56:58.760138 ip-10-0-137-238 kubenswrapper[2529]: I0728 05:56:58.760122 2529 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-qvq6l_c4033dc0-62e3-4974-9d66-c98ea5033f7f/csi-liveness-probe/0.log"