Sep 11 05:18:25.846868 ip-10-0-133-194 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 05:18:25.846875 ip-10-0-133-194 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 05:18:25.846883 ip-10-0-133-194 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 05:18:25.847199 ip-10-0-133-194 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 05:18:37.109790 ip-10-0-133-194 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 05:18:37.109809 ip-10-0-133-194 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 040afdd38c69437f8c2078b94c5bf141 -- Sep 11 05:20:26.778873 ip-10-0-133-194 systemd[1]: Starting Kubernetes Kubelet... Sep 11 05:20:27.161250 ip-10-0-133-194 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 05:20:27.161250 ip-10-0-133-194 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 05:20:27.161250 ip-10-0-133-194 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 05:20:27.161250 ip-10-0-133-194 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 05:20:27.161250 ip-10-0-133-194 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 05:20:27.162041 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.161952 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 05:20:27.167274 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167259 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 05:20:27.167274 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167275 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167278 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167283 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167286 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167289 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167292 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167296 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167299 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167302 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167305 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167307 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167310 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167313 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167316 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167318 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167321 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167323 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167333 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167336 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167340 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 05:20:27.167341 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167342 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167345 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167348 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167350 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167353 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167357 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167360 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167362 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167365 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167368 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167372 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167376 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167378 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167381 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167388 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167391 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167394 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167397 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167400 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 05:20:27.167922 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167402 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167405 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167408 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167410 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167413 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167415 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167418 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167420 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167423 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167426 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167428 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167431 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167434 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167436 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167439 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167443 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167447 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167450 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167452 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167455 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 05:20:27.168385 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167457 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167460 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167463 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167465 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167468 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167471 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167474 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167476 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167479 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167481 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167485 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167487 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167490 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167494 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167497 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167500 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167503 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167506 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167509 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167513 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 05:20:27.168894 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167516 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167519 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167544 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167561 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167564 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167568 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167571 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167574 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167577 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167580 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167582 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167585 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167588 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167590 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167593 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167595 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167598 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167600 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.167603 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168338 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 05:20:27.169413 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168345 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168348 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168351 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168354 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168358 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168363 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168366 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168368 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168371 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168373 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168376 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168378 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168382 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168384 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168389 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168393 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168396 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168399 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168403 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 05:20:27.169918 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168406 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168408 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168411 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168414 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168417 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168419 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168423 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168426 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168430 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168433 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168436 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168438 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168441 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168444 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168446 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168449 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168451 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168454 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168457 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168459 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 05:20:27.170389 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168461 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168464 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168466 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168469 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168471 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168475 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168478 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168481 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168484 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168486 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168489 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168492 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168495 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168497 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168500 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168502 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168504 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168507 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168509 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168512 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 05:20:27.170905 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168514 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168517 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168519 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168522 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168524 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168527 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168529 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168532 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168534 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168537 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168539 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168542 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168544 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168561 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168564 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168567 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168569 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168573 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168576 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168580 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 05:20:27.171400 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168582 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168585 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168587 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168590 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168593 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168596 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168598 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168600 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168603 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168606 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168608 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168611 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168613 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168616 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168618 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168621 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168624 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168626 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168629 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168978 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 05:20:27.171908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168982 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168985 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168988 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168991 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168993 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168996 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.168999 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169001 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169006 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169009 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169012 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169016 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169019 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169022 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169028 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169030 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169033 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169035 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169038 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 05:20:27.172412 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169040 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169043 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169046 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169048 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169051 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169053 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169055 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169058 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169060 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169063 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169065 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169068 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169070 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169073 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169075 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169077 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169080 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169082 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169087 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169090 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 05:20:27.172888 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169093 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169096 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169098 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169101 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169104 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169107 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169110 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169113 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169115 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169118 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169120 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169123 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169125 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169128 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169130 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169132 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169135 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169137 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169140 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 05:20:27.173372 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169142 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169145 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169147 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169150 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169153 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169155 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169157 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169160 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169162 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169165 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169167 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169170 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169172 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169174 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169177 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169180 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169183 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169187 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169190 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169194 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 05:20:27.173897 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169197 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169200 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169202 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169205 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169207 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169210 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169212 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169215 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169217 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169219 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169222 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169225 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169227 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169244 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169250 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169253 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169255 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169258 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169261 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.169263 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 05:20:27.174382 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169339 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169386 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169393 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169397 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169401 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169404 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169414 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169419 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169422 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169426 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169429 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169432 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169437 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169440 2577 flags.go:64] FLAG: --cgroup-root="" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169443 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169446 2577 flags.go:64] FLAG: --client-ca-file="" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169449 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169452 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169456 2577 flags.go:64] FLAG: --cluster-domain="" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169459 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169462 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169465 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169468 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169472 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 05:20:27.174895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169475 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169478 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169483 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169486 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169489 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169492 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169496 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169499 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169502 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169506 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169509 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169512 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169514 2577 flags.go:64] FLAG: --enable-server="true" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169517 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169522 2577 flags.go:64] FLAG: --event-burst="100" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169528 2577 flags.go:64] FLAG: --event-qps="50" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169531 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169534 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169537 2577 flags.go:64] FLAG: --eviction-hard="" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169541 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169544 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169561 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169565 2577 flags.go:64] FLAG: --eviction-soft="" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169568 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169571 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 05:20:27.175487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169574 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169577 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169580 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169583 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169585 2577 flags.go:64] FLAG: --feature-gates="" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169589 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169592 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169595 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169598 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169603 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169606 2577 flags.go:64] FLAG: --help="false" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169609 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169612 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169615 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169618 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169622 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169625 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169628 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169631 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169634 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169637 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169639 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169645 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169648 2577 flags.go:64] FLAG: --kube-reserved="" Sep 11 05:20:27.176136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169651 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169653 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169657 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169659 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169662 2577 flags.go:64] FLAG: --lock-file="" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169666 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169669 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169671 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169677 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169680 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169682 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169685 2577 flags.go:64] FLAG: --logging-format="text" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169688 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169691 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169694 2577 flags.go:64] FLAG: --manifest-url="" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169697 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169701 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169705 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169709 2577 flags.go:64] FLAG: --max-pods="110" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169712 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169715 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169718 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169721 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169724 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169727 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 05:20:27.176734 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169730 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169737 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169740 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169743 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169746 2577 flags.go:64] FLAG: --pod-cidr="" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169751 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169756 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169760 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169763 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169766 2577 flags.go:64] FLAG: --port="10250" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169769 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169771 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d03d1e095b7df3cc" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169775 2577 flags.go:64] FLAG: --qos-reserved="" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169778 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169781 2577 flags.go:64] FLAG: --register-node="true" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169784 2577 flags.go:64] FLAG: --register-with-taints="" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169788 2577 flags.go:64] FLAG: --registry-burst="10" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169790 2577 flags.go:64] FLAG: --registry-qps="5" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169793 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169796 2577 flags.go:64] FLAG: --reserved-memory="" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169800 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169803 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169806 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169809 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169814 2577 flags.go:64] FLAG: --runonce="false" Sep 11 05:20:27.177380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169817 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169820 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169822 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169825 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169828 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169832 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169835 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169838 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169841 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169843 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169846 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169849 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169852 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169856 2577 flags.go:64] FLAG: --system-cgroups="" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169859 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169864 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169867 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169870 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169874 2577 flags.go:64] FLAG: --tls-min-version="" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169877 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169879 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169883 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169886 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169889 2577 flags.go:64] FLAG: --v="2" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169893 2577 flags.go:64] FLAG: --version="false" Sep 11 05:20:27.178011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169896 2577 flags.go:64] FLAG: --vmodule="" Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169900 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.169903 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170019 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170022 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170026 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170030 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170033 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170036 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170039 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170043 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170046 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170049 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170052 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170054 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170057 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170060 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170062 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170065 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170068 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 05:20:27.178657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170072 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170074 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170077 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170080 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170082 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170085 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170087 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170091 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170095 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170098 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170102 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170105 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170109 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170113 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170117 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170120 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170123 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170125 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170129 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 05:20:27.179137 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170132 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170134 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170137 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170140 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170142 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170145 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170147 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170150 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170153 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170155 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170157 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170160 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170163 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170166 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170169 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170172 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170174 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170177 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170180 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170182 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 05:20:27.179628 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170185 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170187 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170190 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170193 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170196 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170199 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170201 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170204 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170206 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170209 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170212 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170216 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170218 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170221 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170223 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170226 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170229 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170231 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170234 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170236 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 05:20:27.180183 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170239 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170241 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170244 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170246 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170249 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170252 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170255 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170258 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170260 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170263 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170266 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170268 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170271 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170273 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170276 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170279 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170282 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170286 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170289 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170292 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 05:20:27.180688 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170295 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170298 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.170301 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.170308 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.177093 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.177109 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.177129 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.177135 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177196 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177202 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177206 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177210 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177213 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177216 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 05:20:27.181339 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177219 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177222 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177225 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177229 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177233 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177236 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177239 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177242 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177244 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177247 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177250 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177252 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177255 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177257 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177261 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177264 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177267 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177270 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177273 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 05:20:27.181726 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177276 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177279 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177281 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177284 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177295 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177298 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177300 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177303 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177306 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177308 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177311 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177313 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177316 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177318 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177320 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177323 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177326 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177328 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177330 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177333 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 05:20:27.182219 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177336 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177338 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177341 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177343 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177345 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177348 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177351 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177353 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177356 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177358 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177361 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177364 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177366 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177368 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177371 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177373 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177377 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177386 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177389 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177392 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177395 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 05:20:27.182948 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177397 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177400 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177402 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177405 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177408 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177410 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177413 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177415 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177418 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177420 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177423 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177425 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177428 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177430 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177433 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177435 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177438 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177440 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177443 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177445 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 05:20:27.183802 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177448 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177450 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177453 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177455 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177458 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177461 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177463 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177466 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177469 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177477 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177480 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177483 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177485 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.177490 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 05:20:27.184540 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177653 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177659 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177662 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177665 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177668 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177671 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177674 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177676 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177679 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177682 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177685 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177688 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177691 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177694 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177696 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177699 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177701 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177704 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177706 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 05:20:27.184908 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177709 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177711 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177714 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177716 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177719 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177721 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177724 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177727 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177729 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177738 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177741 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177744 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177746 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177749 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177751 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177754 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177756 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177759 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177761 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 05:20:27.185401 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177764 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177766 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177769 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177771 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177774 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177776 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177779 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177781 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177784 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177786 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177789 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177791 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177794 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177796 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177799 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177802 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177804 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177807 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177809 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177812 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 05:20:27.185923 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177814 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177817 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177820 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177828 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177830 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177833 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177835 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177838 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177840 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177843 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177845 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177848 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177850 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177853 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177855 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177858 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177860 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177862 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177865 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177867 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 05:20:27.186460 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177870 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177872 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177875 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177877 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177880 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177882 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177885 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177887 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177890 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177893 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177895 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177898 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177901 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177903 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177906 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177908 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177916 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177918 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177922 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 05:20:27.186996 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177932 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 05:20:27.187490 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:27.177935 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 05:20:27.187490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.177940 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 05:20:27.187490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.178057 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 05:20:27.187490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.179928 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 05:20:27.187490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.180803 2577 server.go:1019] "Starting client certificate rotation" Sep 11 05:20:27.187490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.180936 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 05:20:27.187490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.181030 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 05:20:27.187490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.183217 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 05:20:27.188681 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.188657 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 05:20:27.199532 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.199514 2577 log.go:25] "Validated CRI v1 runtime API" Sep 11 05:20:27.204112 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.204094 2577 log.go:25] "Validated CRI v1 image API" Sep 11 05:20:27.205039 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.205024 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 05:20:27.209415 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.209396 2577 fs.go:135] Filesystem UUIDs: map[3901b0d8-515e-4769-ad95-c4457b8e6666:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7faa2eff-4f62-407e-8008-b269c9c43a15:/dev/nvme0n1p3] Sep 11 05:20:27.209512 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.209413 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 05:20:27.214439 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.214326 2577 manager.go:216] Machine: {Timestamp:2026-09-11 05:20:27.213098296 +0000 UTC m=+0.322836454 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099933 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cc157b5b0e245656167a0a21362b8 SystemUUID:ec2cc157-b5b0-e245-6561-67a0a21362b8 BootID:040afdd3-8c69-437f-8c20-78b94c5bf141 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e4:f2:a3:02:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e4:f2:a3:02:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:58:7e:9d:83:14 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 05:20:27.214439 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.214432 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 05:20:27.214589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.214480 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 05:20:27.214589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.214536 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 05:20:27.217237 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.217211 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 05:20:27.217369 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.217240 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 05:20:27.217808 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.217798 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 05:20:27.217838 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.217812 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 05:20:27.217838 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.217820 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 05:20:27.218505 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.218494 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 05:20:27.219145 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.219136 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 05:20:27.222290 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.222279 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 11 05:20:27.222406 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.222397 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 05:20:27.222439 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.222426 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 11 05:20:27.222439 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.222435 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 05:20:27.222502 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.222453 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 05:20:27.222502 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.222461 2577 kubelet.go:398] "Adding apiserver pod source" Sep 11 05:20:27.222502 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.222471 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 05:20:27.222839 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.222828 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 05:20:27.222876 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.222844 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 05:20:27.225853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.225832 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 05:20:27.228798 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.228777 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 05:20:27.228884 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.228825 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 05:20:27.230167 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230149 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 05:20:27.230167 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230169 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230176 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230183 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230195 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230200 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230214 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230219 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230225 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230231 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230240 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 05:20:27.230264 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.230245 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 05:20:27.231320 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.231309 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 05:20:27.231320 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.231320 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 05:20:27.231732 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.231709 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 05:20:27.231799 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.231753 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 05:20:27.234938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.234925 2577 server.go:1268] "Started kubelet" Sep 11 05:20:27.235101 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.235013 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 05:20:27.235153 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.235115 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 05:20:27.235212 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.235123 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 05:20:27.235860 ip-10-0-133-194 systemd[1]: Started Kubernetes Kubelet. Sep 11 05:20:27.237174 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.235861 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 05:20:27.237174 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.236454 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 11 05:20:27.238194 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.237583 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khdq6" Sep 11 05:20:27.239078 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.239049 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 05:20:27.241949 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.241929 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 05:20:27.242044 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.241998 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 05:20:27.242580 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.242566 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 05:20:27.243190 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.242863 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:27.243414 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243399 2577 factory.go:55] Registering systemd factory Sep 11 05:20:27.243482 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243419 2577 factory.go:223] Registration of the systemd container factory successfully Sep 11 05:20:27.243482 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243407 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 05:20:27.243482 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243440 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 05:20:27.243482 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243407 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 05:20:27.243659 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243519 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 11 05:20:27.243659 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243529 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 11 05:20:27.243830 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243812 2577 factory.go:153] Registering CRI-O factory Sep 11 05:20:27.243830 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243831 2577 factory.go:223] Registration of the crio container factory successfully Sep 11 05:20:27.243830 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.243032 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-194.ec2.internal.18d42ca2398e94ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-194.ec2.internal,UID:ip-10-0-133-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-194.ec2.internal,},FirstTimestamp:2026-09-11 05:20:27.23489915 +0000 UTC m=+0.344637307,LastTimestamp:2026-09-11 05:20:27.23489915 +0000 UTC m=+0.344637307,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-194.ec2.internal,}" Sep 11 05:20:27.244051 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243878 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 05:20:27.244051 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243895 2577 factory.go:103] Registering Raw factory Sep 11 05:20:27.244051 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.243910 2577 manager.go:1196] Started watching for new ooms in manager Sep 11 05:20:27.244791 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.244775 2577 manager.go:349] Starting recovery of all containers Sep 11 05:20:27.245756 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.244854 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khdq6" Sep 11 05:20:27.249772 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.249734 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 05:20:27.250003 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.249975 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 05:20:27.256131 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.256113 2577 manager.go:354] Recovery completed Sep 11 05:20:27.260323 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.260310 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 05:20:27.262096 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.262079 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory" Sep 11 05:20:27.262166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.262112 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 05:20:27.262166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.262125 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID" Sep 11 05:20:27.262725 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.262711 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 05:20:27.262725 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.262724 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 05:20:27.262817 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.262738 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 11 05:20:27.262817 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.262768 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 05:20:27.263502 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.263482 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory" Sep 11 05:20:27.263589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.263514 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 05:20:27.263589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.263528 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID" Sep 11 05:20:27.263971 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.263960 2577 policy_none.go:49] "None policy: Start" Sep 11 05:20:27.264005 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.263976 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 05:20:27.264005 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.263986 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 05:20:27.264763 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.264749 2577 policy_none.go:47] "Start" Sep 11 05:20:27.288143 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.288128 2577 manager.go:337] "Starting Device Plugin manager" Sep 11 05:20:27.288234 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.288160 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 05:20:27.288234 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.288169 2577 server.go:87] "Starting device plugin registration server" Sep 11 05:20:27.288461 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.288445 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 05:20:27.288570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.288462 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 05:20:27.288646 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.288639 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 05:20:27.288724 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.288709 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 05:20:27.288724 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.288723 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 05:20:27.289248 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.289228 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 05:20:27.289307 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.289293 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:27.347962 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.347927 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 05:20:27.349936 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.349055 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 05:20:27.349936 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.349108 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 05:20:27.349936 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.349134 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 05:20:27.349936 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.349224 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 05:20:27.351310 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.351295 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 05:20:27.389020 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.389001 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 05:20:27.390666 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.390651 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory" Sep 11 05:20:27.390733 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.390682 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 05:20:27.390733 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.390695 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID" Sep 11 05:20:27.390733 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.390723 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.398471 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.398456 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.398535 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.398477 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-194.ec2.internal\": node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:27.409903 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.409887 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:27.449636 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.449574 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"] Sep 11 05:20:27.449636 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.449630 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 05:20:27.450893 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.450879 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory" Sep 11 05:20:27.450965 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.450906 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 05:20:27.450965 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.450917 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID" Sep 11 05:20:27.453003 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.452992 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 05:20:27.453125 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.453113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.453163 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.453139 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 05:20:27.453747 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.453728 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory" Sep 11 05:20:27.453747 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.453747 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 05:20:27.453836 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.453760 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID" Sep 11 05:20:27.453836 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.453792 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory" Sep 11 05:20:27.453836 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.453809 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 05:20:27.453836 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.453818 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID" Sep 11 05:20:27.455741 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.455729 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.455791 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.455750 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 05:20:27.457074 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.457060 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory" Sep 11 05:20:27.457152 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.457086 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 05:20:27.457152 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.457100 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID" Sep 11 05:20:27.487247 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.487226 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-194.ec2.internal\" not found" node="ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.491499 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.491482 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-194.ec2.internal\" not found" node="ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.510149 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.510129 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:27.544589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.544568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.544667 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.544593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.544667 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.544610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bee55f8a1340335ec8a8808d97d355d4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-194.ec2.internal\" (UID: \"bee55f8a1340335ec8a8808d97d355d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.610569 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.610537 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:27.644838 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.644798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bee55f8a1340335ec8a8808d97d355d4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-194.ec2.internal\" (UID: \"bee55f8a1340335ec8a8808d97d355d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.644838 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.644838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.645006 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.644854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.645006 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.644913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.645006 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.644921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.645006 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.644961 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bee55f8a1340335ec8a8808d97d355d4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-194.ec2.internal\" (UID: \"bee55f8a1340335ec8a8808d97d355d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.711089 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.711034 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:27.789666 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.789622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.794355 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:27.794280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" Sep 11 05:20:27.811620 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.811592 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:27.912127 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:27.912089 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:28.012614 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:28.012588 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:28.064015 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.063984 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 05:20:28.114537 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:28.114512 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:28.181599 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.181572 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 05:20:28.182164 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.181700 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 05:20:28.182164 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.181754 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 05:20:28.215160 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:28.215133 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:28.250886 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.249117 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 05:15:27 +0000 UTC" deadline="2028-06-19 20:08:52.96969022 +0000 UTC" Sep 11 05:20:28.250886 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.249151 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15542h48m24.720543492s" Sep 11 05:20:28.316146 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:28.316101 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:28.393562 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:28.393501 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee55f8a1340335ec8a8808d97d355d4.slice/crio-d4c96b699a3f05243d75eade2b625699e05cf519062c80e5b33daa54c535c26b WatchSource:0}: Error finding container d4c96b699a3f05243d75eade2b625699e05cf519062c80e5b33daa54c535c26b: Status 404 returned error can't find the container with id d4c96b699a3f05243d75eade2b625699e05cf519062c80e5b33daa54c535c26b Sep 11 05:20:28.393851 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:28.393833 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f1e051c1877a9e2bbc49221450c3da.slice/crio-ebd31d894096d71bea15fe56bb9f8d539f80f1e3a59ece3f9e34cd68d022f667 WatchSource:0}: Error finding container ebd31d894096d71bea15fe56bb9f8d539f80f1e3a59ece3f9e34cd68d022f667: Status 404 returned error can't find the container with id ebd31d894096d71bea15fe56bb9f8d539f80f1e3a59ece3f9e34cd68d022f667 Sep 11 05:20:28.400134 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.400118 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 05:20:28.417128 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:28.417103 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:28.521970 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:28.521879 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:28.623389 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:28.623349 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:28.714039 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.714011 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 05:20:28.723760 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:28.723735 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found" Sep 11 05:20:28.763152 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.763125 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 05:20:28.843407 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.843329 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" Sep 11 05:20:28.855225 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.855200 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 05:20:28.856059 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.856046 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" Sep 11 05:20:28.866207 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:28.866182 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 05:20:29.223779 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.223697 2577 apiserver.go:52] "Watching apiserver" Sep 11 05:20:29.231298 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.231269 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 05:20:29.233403 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.233369 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-m7b82","openshift-ovn-kubernetes/ovnkube-node-srxzk","kube-system/global-pull-secret-syncer-5fkvd","kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal","openshift-cluster-node-tuning-operator/tuned-fzhcj","openshift-image-registry/node-ca-mvntc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal","openshift-multus/multus-additional-cni-plugins-2svzc","kube-system/konnectivity-agent-gvw7p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k","openshift-dns/node-resolver-lk5n4","openshift-multus/multus-6th6w","openshift-multus/network-metrics-daemon-hqmnw","openshift-network-diagnostics/network-check-target-g8zdr"] Sep 11 05:20:29.236078 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.236062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:29.236171 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.236133 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:29.238485 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.238465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.240948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.240926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.241046 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.240990 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:29.241112 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.241100 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 05:20:29.241178 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.241118 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8vkv5\"" Sep 11 05:20:29.241232 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.241189 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 05:20:29.241232 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.241219 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 05:20:29.241447 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.241431 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 05:20:29.242370 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.242355 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 05:20:29.242437 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.242366 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 05:20:29.242494 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.242445 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 05:20:29.243286 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.243266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.246143 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.245865 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 05:20:29.246243 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.246143 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 05:20:29.246243 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.246231 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mllwz\"" Sep 11 05:20:29.246387 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.246369 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 05:20:29.247858 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.247426 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.249841 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.249820 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 05:20:29.249933 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.249841 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6nbn2\"" Sep 11 05:20:29.249933 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.249918 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 05:20:29.250053 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.249932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.252085 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.252064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-25bhs\"" Sep 11 05:20:29.252085 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.252086 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 05:20:29.252248 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.252230 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 05:20:29.252393 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.252364 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 05:20:29.252471 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.252446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:29.252529 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.252517 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:29.254894 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.254873 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.255253 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255221 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-ovn-node-metrics-cert\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255350 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-kubernetes\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.255350 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skl6\" (UniqueName: \"kubernetes.io/projected/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-kube-api-access-8skl6\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255350 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a540b042-b24e-416e-b30a-84346d4dd47f-dbus\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.255350 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255312 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 05:20:29.255350 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-run-ovn\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255647 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255378 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-run-netns\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255647 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-node-log\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255647 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-log-socket\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255647 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-sysctl-d\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.255647 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:29.255647 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-kubelet\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255647 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255612 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1f7ce08-a06a-4063-92e7-8f2c02044ba8-iptables-alerter-script\") pod \"iptables-alerter-m7b82\" (UID: \"d1f7ce08-a06a-4063-92e7-8f2c02044ba8\") " pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.255647 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255639 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-sys\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-lib-modules\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255683 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-host\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-tuned\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-tmp\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-etc-openvswitch\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-run-openvswitch\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-var-lib-openvswitch\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-cni-bin\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-systemd\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-var-lib-kubelet\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255900 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-slash\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-ovnkube-script-lib\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1f7ce08-a06a-4063-92e7-8f2c02044ba8-host-slash\") pod \"iptables-alerter-m7b82\" (UID: \"d1f7ce08-a06a-4063-92e7-8f2c02044ba8\") " pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.255964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.255961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-run\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9vq\" (UniqueName: \"kubernetes.io/projected/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-kube-api-access-4t9vq\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-cni-netd\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a540b042-b24e-416e-b30a-84346d4dd47f-kubelet-config\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq4mc\" (UniqueName: \"kubernetes.io/projected/ac746c00-7462-4686-9f96-7d5be8804348-kube-api-access-hq4mc\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-systemd-units\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-run-ovn-kubernetes\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-sysconfig\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-run-systemd\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrmvg\" (UniqueName: \"kubernetes.io/projected/d1f7ce08-a06a-4063-92e7-8f2c02044ba8-kube-api-access-vrmvg\") pod \"iptables-alerter-m7b82\" (UID: \"d1f7ce08-a06a-4063-92e7-8f2c02044ba8\") " pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256254 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-modprobe-d\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-sysctl-conf\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-ovnkube-config\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.256420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.256351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-env-overrides\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.257332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.257317 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 05:20:29.257400 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.257339 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 05:20:29.257463 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.257396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:29.258846 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.258829 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 05:20:29.258846 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.258836 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 05:20:29.259006 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.258832 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k5h7m\"" Sep 11 05:20:29.259071 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.258832 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 05:20:29.259827 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.259804 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 05:20:29.259827 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.259822 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p6wfc\"" Sep 11 05:20:29.259827 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.259826 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 05:20:29.260396 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.260382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.262743 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.262726 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 05:20:29.262843 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.262783 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 05:20:29.262843 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.262817 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 05:20:29.262965 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.262846 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d5l68\"" Sep 11 05:20:29.263019 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.263012 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.265174 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.265154 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 05:20:29.265346 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.265324 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 05:20:29.265454 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.265441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.265539 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.265524 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jxvgs\"" Sep 11 05:20:29.267921 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.267904 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 05:20:29.268011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.267993 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wsvbc\"" Sep 11 05:20:29.283763 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.283740 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tt4sj" Sep 11 05:20:29.291207 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.291186 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tt4sj" Sep 11 05:20:29.344475 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.344456 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 05:20:29.353756 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.353704 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" event={"ID":"02f1e051c1877a9e2bbc49221450c3da","Type":"ContainerStarted","Data":"ebd31d894096d71bea15fe56bb9f8d539f80f1e3a59ece3f9e34cd68d022f667"} Sep 11 05:20:29.354926 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.354905 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" event={"ID":"bee55f8a1340335ec8a8808d97d355d4","Type":"ContainerStarted","Data":"d4c96b699a3f05243d75eade2b625699e05cf519062c80e5b33daa54c535c26b"} Sep 11 05:20:29.356584 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-modprobe-d\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.356693 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-run-netns\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.356693 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t4gs\" (UniqueName: \"kubernetes.io/projected/830c602d-f84b-4211-8203-78d39e810f0a-kube-api-access-9t4gs\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.356788 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ad2bdd1-8798-4983-9878-0060b20d0e3e-konnectivity-ca\") pod \"konnectivity-agent-gvw7p\" (UID: \"4ad2bdd1-8798-4983-9878-0060b20d0e3e\") " pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:29.356788 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-ovnkube-config\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.356884 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-env-overrides\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.356884 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a540b042-b24e-416e-b30a-84346d4dd47f-dbus\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.356884 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356825 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-modprobe-d\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.357044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a540b042-b24e-416e-b30a-84346d4dd47f-dbus\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.357044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-run-ovn\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.357044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-run-netns\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.357044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.356998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-log-socket\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.357044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-sysctl-d\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-run-ovn\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-sys\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-host\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-sys\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357142 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-run-netns\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-kubelet\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357174 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-kubelet\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1f7ce08-a06a-4063-92e7-8f2c02044ba8-iptables-alerter-script\") pod \"iptables-alerter-m7b82\" (UID: \"d1f7ce08-a06a-4063-92e7-8f2c02044ba8\") " pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-tmp\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2010dca1-86fb-471b-8a76-b9ff03a30d37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hshl\" (UniqueName: \"kubernetes.io/projected/2010dca1-86fb-471b-8a76-b9ff03a30d37-kube-api-access-6hshl\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-etc-openvswitch\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-ovnkube-config\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.357332 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-run-openvswitch\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-systemd\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357349 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-env-overrides\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-var-lib-cni-multus\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-socket-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357406 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-host\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-var-lib-kubelet\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-ovnkube-script-lib\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357502 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-run-openvswitch\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-etc-openvswitch\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ad2bdd1-8798-4983-9878-0060b20d0e3e-agent-certs\") pod \"konnectivity-agent-gvw7p\" (UID: \"4ad2bdd1-8798-4983-9878-0060b20d0e3e\") " pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-log-socket\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a540b042-b24e-416e-b30a-84346d4dd47f-kubelet-config\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq4mc\" (UniqueName: \"kubernetes.io/projected/ac746c00-7462-4686-9f96-7d5be8804348-kube-api-access-hq4mc\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357645 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-sysctl-d\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-cnibin\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.357686 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357696 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a540b042-b24e-416e-b30a-84346d4dd47f-kubelet-config\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.358046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-multus-conf-dir\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.357778 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs podName:ac746c00-7462-4686-9f96-7d5be8804348 nodeName:}" failed. No retries permitted until 2026-09-11 05:20:29.857740803 +0000 UTC m=+2.967478950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs") pod "network-metrics-daemon-hqmnw" (UID: "ac746c00-7462-4686-9f96-7d5be8804348") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1f7ce08-a06a-4063-92e7-8f2c02044ba8-iptables-alerter-script\") pod \"iptables-alerter-m7b82\" (UID: \"d1f7ce08-a06a-4063-92e7-8f2c02044ba8\") " pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357827 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357897 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-systemd\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-run-multus-certs\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.357992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-etc-selinux\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2010dca1-86fb-471b-8a76-b9ff03a30d37-cni-binary-copy\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3fa8974b-7bf1-4d99-884d-d5e923f8e568-hosts-file\") pod \"node-resolver-lk5n4\" (UID: \"3fa8974b-7bf1-4d99-884d-d5e923f8e568\") " pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-sysconfig\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/830c602d-f84b-4211-8203-78d39e810f0a-multus-daemon-config\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-device-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-ovnkube-script-lib\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2010dca1-86fb-471b-8a76-b9ff03a30d37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-sysconfig\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-run-systemd\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.358852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrmvg\" (UniqueName: \"kubernetes.io/projected/d1f7ce08-a06a-4063-92e7-8f2c02044ba8-kube-api-access-vrmvg\") pod \"iptables-alerter-m7b82\" (UID: \"d1f7ce08-a06a-4063-92e7-8f2c02044ba8\") " pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-run-systemd\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-sysctl-conf\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-ovn-node-metrics-cert\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-kubernetes\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skl6\" (UniqueName: \"kubernetes.io/projected/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-kube-api-access-8skl6\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-node-log\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358408 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-sys-fs\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358476 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-sysctl-conf\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvt4\" (UniqueName: \"kubernetes.io/projected/d4d6f060-e767-49e2-8c41-a32d0b1a83ae-kube-api-access-dcvt4\") pod \"node-ca-mvntc\" (UID: \"d4d6f060-e767-49e2-8c41-a32d0b1a83ae\") " pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358524 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-node-log\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-kubernetes\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-var-lib-cni-bin\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-lib-modules\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.359726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-tuned\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-etc-kubernetes\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-lib-modules\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-registration-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2010dca1-86fb-471b-8a76-b9ff03a30d37-system-cni-dir\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-var-lib-openvswitch\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-cni-bin\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-var-lib-kubelet\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.358993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-multus-socket-dir-parent\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-var-lib-openvswitch\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-run-k8s-cni-cncf-io\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4h2z\" (UniqueName: \"kubernetes.io/projected/130e718b-93e4-4541-9a2a-9ee22ac205de-kube-api-access-l4h2z\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359065 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-var-lib-kubelet\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4d6f060-e767-49e2-8c41-a32d0b1a83ae-serviceca\") pod \"node-ca-mvntc\" (UID: \"d4d6f060-e767-49e2-8c41-a32d0b1a83ae\") " pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-slash\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359108 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-cni-bin\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1f7ce08-a06a-4063-92e7-8f2c02044ba8-host-slash\") pod \"iptables-alerter-m7b82\" (UID: \"d1f7ce08-a06a-4063-92e7-8f2c02044ba8\") " pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.360570 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-run\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9vq\" (UniqueName: \"kubernetes.io/projected/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-kube-api-access-4t9vq\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359186 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-slash\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/830c602d-f84b-4211-8203-78d39e810f0a-cni-binary-copy\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359246 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-run\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359246 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa8974b-7bf1-4d99-884d-d5e923f8e568-tmp-dir\") pod \"node-resolver-lk5n4\" (UID: \"3fa8974b-7bf1-4d99-884d-d5e923f8e568\") " pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh4p5\" (UniqueName: \"kubernetes.io/projected/3fa8974b-7bf1-4d99-884d-d5e923f8e568-kube-api-access-hh4p5\") pod \"node-resolver-lk5n4\" (UID: \"3fa8974b-7bf1-4d99-884d-d5e923f8e568\") " pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-cni-netd\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-systemd-units\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-run-ovn-kubernetes\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-multus-cni-dir\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2010dca1-86fb-471b-8a76-b9ff03a30d37-cnibin\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359501 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1f7ce08-a06a-4063-92e7-8f2c02044ba8-host-slash\") pod \"iptables-alerter-m7b82\" (UID: \"d1f7ce08-a06a-4063-92e7-8f2c02044ba8\") " pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359548 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-cni-netd\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-system-cni-dir\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-hostroot\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.361529 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.359644 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:29.362442 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4d6f060-e767-49e2-8c41-a32d0b1a83ae-host\") pod \"node-ca-mvntc\" (UID: \"d4d6f060-e767-49e2-8c41-a32d0b1a83ae\") " pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.362442 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-host-run-ovn-kubernetes\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.362442 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359725 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-systemd-units\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.362442 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.359753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2010dca1-86fb-471b-8a76-b9ff03a30d37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.362442 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.359784 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret podName:a540b042-b24e-416e-b30a-84346d4dd47f nodeName:}" failed. No retries permitted until 2026-09-11 05:20:29.859764695 +0000 UTC m=+2.969503062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret") pod "global-pull-secret-syncer-5fkvd" (UID: "a540b042-b24e-416e-b30a-84346d4dd47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:29.362442 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.361520 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-tmp\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.362442 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.361603 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-etc-tuned\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.362442 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.361848 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-ovn-node-metrics-cert\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.369099 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.369070 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrmvg\" (UniqueName: \"kubernetes.io/projected/d1f7ce08-a06a-4063-92e7-8f2c02044ba8-kube-api-access-vrmvg\") pod \"iptables-alerter-m7b82\" (UID: \"d1f7ce08-a06a-4063-92e7-8f2c02044ba8\") " pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.369519 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.369493 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq4mc\" (UniqueName: \"kubernetes.io/projected/ac746c00-7462-4686-9f96-7d5be8804348-kube-api-access-hq4mc\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:29.370093 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.370074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9vq\" (UniqueName: \"kubernetes.io/projected/05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95-kube-api-access-4t9vq\") pod \"tuned-fzhcj\" (UID: \"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95\") " pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.370243 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.370213 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skl6\" (UniqueName: \"kubernetes.io/projected/f8b8a658-19bf-4a12-9a6a-e348e106b6d3-kube-api-access-8skl6\") pod \"ovnkube-node-srxzk\" (UID: \"f8b8a658-19bf-4a12-9a6a-e348e106b6d3\") " pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.460131 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-etc-selinux\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.460298 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2010dca1-86fb-471b-8a76-b9ff03a30d37-cni-binary-copy\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.460298 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3fa8974b-7bf1-4d99-884d-d5e923f8e568-hosts-file\") pod \"node-resolver-lk5n4\" (UID: \"3fa8974b-7bf1-4d99-884d-d5e923f8e568\") " pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.460298 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/830c602d-f84b-4211-8203-78d39e810f0a-multus-daemon-config\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.460298 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-device-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.460298 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2010dca1-86fb-471b-8a76-b9ff03a30d37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.460298 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-etc-selinux\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-sys-fs\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvt4\" (UniqueName: \"kubernetes.io/projected/d4d6f060-e767-49e2-8c41-a32d0b1a83ae-kube-api-access-dcvt4\") pod \"node-ca-mvntc\" (UID: \"d4d6f060-e767-49e2-8c41-a32d0b1a83ae\") " pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-device-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3fa8974b-7bf1-4d99-884d-d5e923f8e568-hosts-file\") pod \"node-resolver-lk5n4\" (UID: \"3fa8974b-7bf1-4d99-884d-d5e923f8e568\") " pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-var-lib-cni-bin\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-etc-kubernetes\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-registration-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-sys-fs\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2010dca1-86fb-471b-8a76-b9ff03a30d37-system-cni-dir\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-multus-socket-dir-parent\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-run-k8s-cni-cncf-io\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4h2z\" (UniqueName: \"kubernetes.io/projected/130e718b-93e4-4541-9a2a-9ee22ac205de-kube-api-access-l4h2z\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-registration-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4d6f060-e767-49e2-8c41-a32d0b1a83ae-serviceca\") pod \"node-ca-mvntc\" (UID: \"d4d6f060-e767-49e2-8c41-a32d0b1a83ae\") " pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460685 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-etc-kubernetes\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.460710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-var-lib-cni-bin\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/830c602d-f84b-4211-8203-78d39e810f0a-cni-binary-copy\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-run-k8s-cni-cncf-io\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460741 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2010dca1-86fb-471b-8a76-b9ff03a30d37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa8974b-7bf1-4d99-884d-d5e923f8e568-tmp-dir\") pod \"node-resolver-lk5n4\" (UID: \"3fa8974b-7bf1-4d99-884d-d5e923f8e568\") " pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh4p5\" (UniqueName: \"kubernetes.io/projected/3fa8974b-7bf1-4d99-884d-d5e923f8e568-kube-api-access-hh4p5\") pod \"node-resolver-lk5n4\" (UID: \"3fa8974b-7bf1-4d99-884d-d5e923f8e568\") " pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460796 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2010dca1-86fb-471b-8a76-b9ff03a30d37-system-cni-dir\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460837 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-multus-cni-dir\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460849 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-multus-socket-dir-parent\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2010dca1-86fb-471b-8a76-b9ff03a30d37-cnibin\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460868 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2010dca1-86fb-471b-8a76-b9ff03a30d37-cni-binary-copy\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/830c602d-f84b-4211-8203-78d39e810f0a-multus-daemon-config\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-system-cni-dir\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460909 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2010dca1-86fb-471b-8a76-b9ff03a30d37-cnibin\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460933 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-hostroot\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4d6f060-e767-49e2-8c41-a32d0b1a83ae-host\") pod \"node-ca-mvntc\" (UID: \"d4d6f060-e767-49e2-8c41-a32d0b1a83ae\") " pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.461531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460979 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-multus-cni-dir\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.460981 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-system-cni-dir\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461017 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-hostroot\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461028 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4d6f060-e767-49e2-8c41-a32d0b1a83ae-host\") pod \"node-ca-mvntc\" (UID: \"d4d6f060-e767-49e2-8c41-a32d0b1a83ae\") " pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2010dca1-86fb-471b-8a76-b9ff03a30d37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-run-netns\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t4gs\" (UniqueName: \"kubernetes.io/projected/830c602d-f84b-4211-8203-78d39e810f0a-kube-api-access-9t4gs\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ad2bdd1-8798-4983-9878-0060b20d0e3e-konnectivity-ca\") pod \"konnectivity-agent-gvw7p\" (UID: \"4ad2bdd1-8798-4983-9878-0060b20d0e3e\") " pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa8974b-7bf1-4d99-884d-d5e923f8e568-tmp-dir\") pod \"node-resolver-lk5n4\" (UID: \"3fa8974b-7bf1-4d99-884d-d5e923f8e568\") " pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4d6f060-e767-49e2-8c41-a32d0b1a83ae-serviceca\") pod \"node-ca-mvntc\" (UID: \"d4d6f060-e767-49e2-8c41-a32d0b1a83ae\") " pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2010dca1-86fb-471b-8a76-b9ff03a30d37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hshl\" (UniqueName: \"kubernetes.io/projected/2010dca1-86fb-471b-8a76-b9ff03a30d37-kube-api-access-6hshl\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.461998 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/830c602d-f84b-4211-8203-78d39e810f0a-cni-binary-copy\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-var-lib-cni-multus\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-run-netns\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-socket-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-var-lib-kubelet\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.462368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2010dca1-86fb-471b-8a76-b9ff03a30d37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462138 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ad2bdd1-8798-4983-9878-0060b20d0e3e-konnectivity-ca\") pod \"konnectivity-agent-gvw7p\" (UID: \"4ad2bdd1-8798-4983-9878-0060b20d0e3e\") " pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-var-lib-cni-multus\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-var-lib-kubelet\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ad2bdd1-8798-4983-9878-0060b20d0e3e-agent-certs\") pod \"konnectivity-agent-gvw7p\" (UID: \"4ad2bdd1-8798-4983-9878-0060b20d0e3e\") " pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462374 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/130e718b-93e4-4541-9a2a-9ee22ac205de-socket-dir\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462391 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-cnibin\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-cnibin\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-multus-conf-dir\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-run-multus-certs\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462697 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2010dca1-86fb-471b-8a76-b9ff03a30d37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.462855 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-multus-conf-dir\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.463168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.463049 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/830c602d-f84b-4211-8203-78d39e810f0a-host-run-multus-certs\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.467242 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.467219 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 05:20:29.467242 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.467240 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 05:20:29.467417 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.467251 2577 projected.go:196] Error preparing data for projected volume kube-api-access-2hrsf for pod openshift-network-diagnostics/network-check-target-g8zdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:29.467417 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.467307 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf podName:e699639b-b57e-4d9a-93b9-ec9b71a507fc nodeName:}" failed. No retries permitted until 2026-09-11 05:20:29.967294842 +0000 UTC m=+3.077032987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2hrsf" (UniqueName: "kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf") pod "network-check-target-g8zdr" (UID: "e699639b-b57e-4d9a-93b9-ec9b71a507fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:29.467846 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.467819 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ad2bdd1-8798-4983-9878-0060b20d0e3e-agent-certs\") pod \"konnectivity-agent-gvw7p\" (UID: \"4ad2bdd1-8798-4983-9878-0060b20d0e3e\") " pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:29.469769 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.469741 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvt4\" (UniqueName: \"kubernetes.io/projected/d4d6f060-e767-49e2-8c41-a32d0b1a83ae-kube-api-access-dcvt4\") pod \"node-ca-mvntc\" (UID: \"d4d6f060-e767-49e2-8c41-a32d0b1a83ae\") " pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.470452 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.470429 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t4gs\" (UniqueName: \"kubernetes.io/projected/830c602d-f84b-4211-8203-78d39e810f0a-kube-api-access-9t4gs\") pod \"multus-6th6w\" (UID: \"830c602d-f84b-4211-8203-78d39e810f0a\") " pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.470661 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.470611 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh4p5\" (UniqueName: \"kubernetes.io/projected/3fa8974b-7bf1-4d99-884d-d5e923f8e568-kube-api-access-hh4p5\") pod \"node-resolver-lk5n4\" (UID: \"3fa8974b-7bf1-4d99-884d-d5e923f8e568\") " pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.470966 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.470919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4h2z\" (UniqueName: \"kubernetes.io/projected/130e718b-93e4-4541-9a2a-9ee22ac205de-kube-api-access-l4h2z\") pod \"aws-ebs-csi-driver-node-8bp9k\" (UID: \"130e718b-93e4-4541-9a2a-9ee22ac205de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.471070 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.471023 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hshl\" (UniqueName: \"kubernetes.io/projected/2010dca1-86fb-471b-8a76-b9ff03a30d37-kube-api-access-6hshl\") pod \"multus-additional-cni-plugins-2svzc\" (UID: \"2010dca1-86fb-471b-8a76-b9ff03a30d37\") " pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.550795 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.550717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:29.558367 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.558346 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m7b82" Sep 11 05:20:29.566460 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.566436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" Sep 11 05:20:29.571392 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.571375 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvntc" Sep 11 05:20:29.577103 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.577080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2svzc" Sep 11 05:20:29.582945 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.582924 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:29.589821 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.589804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" Sep 11 05:20:29.597662 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.597647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lk5n4" Sep 11 05:20:29.603627 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.603609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6th6w" Sep 11 05:20:29.687951 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.687921 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 05:20:29.696003 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:29.695975 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130e718b_93e4_4541_9a2a_9ee22ac205de.slice/crio-e469f6665763d113146064f653bcaaa7f89c4b6c3fd148b1a88ffd9235d184f6 WatchSource:0}: Error finding container e469f6665763d113146064f653bcaaa7f89c4b6c3fd148b1a88ffd9235d184f6: Status 404 returned error can't find the container with id e469f6665763d113146064f653bcaaa7f89c4b6c3fd148b1a88ffd9235d184f6 Sep 11 05:20:29.699180 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:29.699109 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa8974b_7bf1_4d99_884d_d5e923f8e568.slice/crio-585d37188957b4fba990211eaea8e6fead4d68ab923659e7869d972cb787f26a WatchSource:0}: Error finding container 585d37188957b4fba990211eaea8e6fead4d68ab923659e7869d972cb787f26a: Status 404 returned error can't find the container with id 585d37188957b4fba990211eaea8e6fead4d68ab923659e7869d972cb787f26a Sep 11 05:20:29.701742 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:29.701548 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830c602d_f84b_4211_8203_78d39e810f0a.slice/crio-c29cd78d55aa619f6cecbc187e489298bac42aec4c96c39f109af5c6ece89b46 WatchSource:0}: Error finding container c29cd78d55aa619f6cecbc187e489298bac42aec4c96c39f109af5c6ece89b46: Status 404 returned error can't find the container with id c29cd78d55aa619f6cecbc187e489298bac42aec4c96c39f109af5c6ece89b46 Sep 11 05:20:29.702265 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:29.702228 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f7ce08_a06a_4063_92e7_8f2c02044ba8.slice/crio-670afd6e10c9eb9aff1a1a719b29e25ee8aca28f4897e8642f43d726fc979bfd WatchSource:0}: Error finding container 670afd6e10c9eb9aff1a1a719b29e25ee8aca28f4897e8642f43d726fc979bfd: Status 404 returned error can't find the container with id 670afd6e10c9eb9aff1a1a719b29e25ee8aca28f4897e8642f43d726fc979bfd Sep 11 05:20:29.703184 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:29.703138 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad2bdd1_8798_4983_9878_0060b20d0e3e.slice/crio-4d886394c01334ff07513a8e9994377785405a311b8e34ebe4e75e0b8f8c4dc1 WatchSource:0}: Error finding container 4d886394c01334ff07513a8e9994377785405a311b8e34ebe4e75e0b8f8c4dc1: Status 404 returned error can't find the container with id 4d886394c01334ff07513a8e9994377785405a311b8e34ebe4e75e0b8f8c4dc1 Sep 11 05:20:29.704930 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:29.704903 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2010dca1_86fb_471b_8a76_b9ff03a30d37.slice/crio-2f94d141a2a2ccc9ea4f60ad4a9430c2c17b40a792e3c7d8e722e1aca1a218c4 WatchSource:0}: Error finding container 2f94d141a2a2ccc9ea4f60ad4a9430c2c17b40a792e3c7d8e722e1aca1a218c4: Status 404 returned error can't find the container with id 2f94d141a2a2ccc9ea4f60ad4a9430c2c17b40a792e3c7d8e722e1aca1a218c4 Sep 11 05:20:29.705128 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:29.705101 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d6f060_e767_49e2_8c41_a32d0b1a83ae.slice/crio-c2977a6bcc44d7771f1a6a71aad83762b4c821c250ed2bb3e3647e46b4022c8d WatchSource:0}: Error finding container c2977a6bcc44d7771f1a6a71aad83762b4c821c250ed2bb3e3647e46b4022c8d: Status 404 returned error can't find the container with id c2977a6bcc44d7771f1a6a71aad83762b4c821c250ed2bb3e3647e46b4022c8d Sep 11 05:20:29.706561 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:29.706541 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b8a658_19bf_4a12_9a6a_e348e106b6d3.slice/crio-fc7d9368f8ef3e95314c9ba92e516cff324bd71e54c84a78fb93c25e5d7ecc1e WatchSource:0}: Error finding container fc7d9368f8ef3e95314c9ba92e516cff324bd71e54c84a78fb93c25e5d7ecc1e: Status 404 returned error can't find the container with id fc7d9368f8ef3e95314c9ba92e516cff324bd71e54c84a78fb93c25e5d7ecc1e Sep 11 05:20:29.707252 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:20:29.707233 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a69a1a_eb1d_4eaa_9ab0_138e8b11bb95.slice/crio-6e879caf863d8c5bc115ef05206abf53319a515f33c36b708f8110c0adee0b00 WatchSource:0}: Error finding container 6e879caf863d8c5bc115ef05206abf53319a515f33c36b708f8110c0adee0b00: Status 404 returned error can't find the container with id 6e879caf863d8c5bc115ef05206abf53319a515f33c36b708f8110c0adee0b00 Sep 11 05:20:29.866007 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.865921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:29.866007 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:29.865970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:29.866223 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.866069 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:29.866223 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.866134 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret podName:a540b042-b24e-416e-b30a-84346d4dd47f nodeName:}" failed. No retries permitted until 2026-09-11 05:20:30.866118037 +0000 UTC m=+3.975856197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret") pod "global-pull-secret-syncer-5fkvd" (UID: "a540b042-b24e-416e-b30a-84346d4dd47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:29.866223 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.866074 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:29.866223 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:29.866224 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs podName:ac746c00-7462-4686-9f96-7d5be8804348 nodeName:}" failed. No retries permitted until 2026-09-11 05:20:30.866206909 +0000 UTC m=+3.975945057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs") pod "network-metrics-daemon-hqmnw" (UID: "ac746c00-7462-4686-9f96-7d5be8804348") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:30.068168 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.068142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:30.068307 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:30.068292 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 05:20:30.068358 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:30.068312 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 05:20:30.068358 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:30.068322 2577 projected.go:196] Error preparing data for projected volume kube-api-access-2hrsf for pod openshift-network-diagnostics/network-check-target-g8zdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:30.068450 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:30.068370 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf podName:e699639b-b57e-4d9a-93b9-ec9b71a507fc nodeName:}" failed. No retries permitted until 2026-09-11 05:20:31.068354409 +0000 UTC m=+4.178092564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hrsf" (UniqueName: "kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf") pod "network-check-target-g8zdr" (UID: "e699639b-b57e-4d9a-93b9-ec9b71a507fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:30.294977 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.294874 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 05:15:29 +0000 UTC" deadline="2028-05-03 08:34:47.01818174 +0000 UTC" Sep 11 05:20:30.294977 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.294914 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14403h14m16.723273255s" Sep 11 05:20:30.349425 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.349393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:30.349584 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:30.349530 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:30.361871 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.361807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lk5n4" event={"ID":"3fa8974b-7bf1-4d99-884d-d5e923f8e568","Type":"ContainerStarted","Data":"585d37188957b4fba990211eaea8e6fead4d68ab923659e7869d972cb787f26a"} Sep 11 05:20:30.364316 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.364217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" event={"ID":"130e718b-93e4-4541-9a2a-9ee22ac205de","Type":"ContainerStarted","Data":"e469f6665763d113146064f653bcaaa7f89c4b6c3fd148b1a88ffd9235d184f6"} Sep 11 05:20:30.369893 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.369863 2577 generic.go:356] "Generic (PLEG): container finished" podID="02f1e051c1877a9e2bbc49221450c3da" containerID="4b825e2e01f96b0e1b995fd68023a12f4366bee68ef071122144d19e629047dd" exitCode=0 Sep 11 05:20:30.370755 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.370732 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" event={"ID":"02f1e051c1877a9e2bbc49221450c3da","Type":"ContainerDied","Data":"4b825e2e01f96b0e1b995fd68023a12f4366bee68ef071122144d19e629047dd"} Sep 11 05:20:30.377269 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.377243 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" event={"ID":"f8b8a658-19bf-4a12-9a6a-e348e106b6d3","Type":"ContainerStarted","Data":"fc7d9368f8ef3e95314c9ba92e516cff324bd71e54c84a78fb93c25e5d7ecc1e"} Sep 11 05:20:30.383978 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.383951 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvntc" event={"ID":"d4d6f060-e767-49e2-8c41-a32d0b1a83ae","Type":"ContainerStarted","Data":"c2977a6bcc44d7771f1a6a71aad83762b4c821c250ed2bb3e3647e46b4022c8d"} Sep 11 05:20:30.395708 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.395672 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gvw7p" event={"ID":"4ad2bdd1-8798-4983-9878-0060b20d0e3e","Type":"ContainerStarted","Data":"4d886394c01334ff07513a8e9994377785405a311b8e34ebe4e75e0b8f8c4dc1"} Sep 11 05:20:30.400238 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.400211 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m7b82" event={"ID":"d1f7ce08-a06a-4063-92e7-8f2c02044ba8","Type":"ContainerStarted","Data":"670afd6e10c9eb9aff1a1a719b29e25ee8aca28f4897e8642f43d726fc979bfd"} Sep 11 05:20:30.414001 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.413973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6th6w" event={"ID":"830c602d-f84b-4211-8203-78d39e810f0a","Type":"ContainerStarted","Data":"c29cd78d55aa619f6cecbc187e489298bac42aec4c96c39f109af5c6ece89b46"} Sep 11 05:20:30.416400 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.416375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" event={"ID":"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95","Type":"ContainerStarted","Data":"6e879caf863d8c5bc115ef05206abf53319a515f33c36b708f8110c0adee0b00"} Sep 11 05:20:30.420958 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.420933 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2svzc" event={"ID":"2010dca1-86fb-471b-8a76-b9ff03a30d37","Type":"ContainerStarted","Data":"2f94d141a2a2ccc9ea4f60ad4a9430c2c17b40a792e3c7d8e722e1aca1a218c4"} Sep 11 05:20:30.818532 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.818442 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 05:20:30.875230 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.875193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:30.875402 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:30.875259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:30.875402 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:30.875386 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:30.875516 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:30.875448 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs podName:ac746c00-7462-4686-9f96-7d5be8804348 nodeName:}" failed. No retries permitted until 2026-09-11 05:20:32.875429346 +0000 UTC m=+5.985167508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs") pod "network-metrics-daemon-hqmnw" (UID: "ac746c00-7462-4686-9f96-7d5be8804348") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:30.875877 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:30.875856 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:30.875981 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:30.875968 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret podName:a540b042-b24e-416e-b30a-84346d4dd47f nodeName:}" failed. No retries permitted until 2026-09-11 05:20:32.875902181 +0000 UTC m=+5.985640330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret") pod "global-pull-secret-syncer-5fkvd" (UID: "a540b042-b24e-416e-b30a-84346d4dd47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:31.076999 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:31.076896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:31.077200 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:31.077071 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 05:20:31.077200 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:31.077094 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 05:20:31.077200 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:31.077107 2577 projected.go:196] Error preparing data for projected volume kube-api-access-2hrsf for pod openshift-network-diagnostics/network-check-target-g8zdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:31.077200 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:31.077171 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf podName:e699639b-b57e-4d9a-93b9-ec9b71a507fc nodeName:}" failed. No retries permitted until 2026-09-11 05:20:33.077151769 +0000 UTC m=+6.186889932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hrsf" (UniqueName: "kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf") pod "network-check-target-g8zdr" (UID: "e699639b-b57e-4d9a-93b9-ec9b71a507fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:31.295842 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:31.295788 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 05:15:29 +0000 UTC" deadline="2028-06-03 21:24:32.654262074 +0000 UTC" Sep 11 05:20:31.295842 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:31.295824 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15160h4m1.358442151s" Sep 11 05:20:31.352939 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:31.349681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:31.352939 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:31.349835 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:31.358122 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:31.358088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:31.358537 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:31.358509 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:31.445295 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:31.445255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" event={"ID":"02f1e051c1877a9e2bbc49221450c3da","Type":"ContainerStarted","Data":"dc9c6ef88077389f2f87c0632257e094ff1c25157a922b1e678902400ef701b1"} Sep 11 05:20:31.462840 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:31.462065 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" podStartSLOduration=2.462044288 podStartE2EDuration="2.462044288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:31.461365286 +0000 UTC m=+4.571103458" watchObservedRunningTime="2026-09-11 05:20:31.462044288 +0000 UTC m=+4.571782457" Sep 11 05:20:32.350202 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:32.349655 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:32.350202 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:32.349797 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:32.895503 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:32.895461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:32.895723 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:32.895572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:32.895790 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:32.895737 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:32.895862 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:32.895804 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret podName:a540b042-b24e-416e-b30a-84346d4dd47f nodeName:}" failed. No retries permitted until 2026-09-11 05:20:36.895784854 +0000 UTC m=+10.005523025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret") pod "global-pull-secret-syncer-5fkvd" (UID: "a540b042-b24e-416e-b30a-84346d4dd47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:32.896525 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:32.896224 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:32.896525 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:32.896281 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs podName:ac746c00-7462-4686-9f96-7d5be8804348 nodeName:}" failed. No retries permitted until 2026-09-11 05:20:36.896265318 +0000 UTC m=+10.006003467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs") pod "network-metrics-daemon-hqmnw" (UID: "ac746c00-7462-4686-9f96-7d5be8804348") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:33.097954 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:33.097292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:33.097954 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:33.097487 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 05:20:33.097954 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:33.097509 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 05:20:33.097954 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:33.097521 2577 projected.go:196] Error preparing data for projected volume kube-api-access-2hrsf for pod openshift-network-diagnostics/network-check-target-g8zdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:33.097954 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:33.097581 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf podName:e699639b-b57e-4d9a-93b9-ec9b71a507fc nodeName:}" failed. No retries permitted until 2026-09-11 05:20:37.097561517 +0000 UTC m=+10.207299680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hrsf" (UniqueName: "kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf") pod "network-check-target-g8zdr" (UID: "e699639b-b57e-4d9a-93b9-ec9b71a507fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:33.350462 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:33.350380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:33.350954 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:33.350532 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:33.350954 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:33.350929 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:33.351079 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:33.351054 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:34.349831 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:34.349789 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:34.350035 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:34.349915 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:35.350362 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:35.349841 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:35.350362 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:35.350000 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:35.350362 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:35.350023 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:35.350362 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:35.350137 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:36.349959 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:36.349698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:36.349959 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:36.349835 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:36.929485 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:36.929448 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:36.929962 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:36.929511 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:36.929962 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:36.929651 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:36.929962 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:36.929716 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs podName:ac746c00-7462-4686-9f96-7d5be8804348 nodeName:}" failed. No retries permitted until 2026-09-11 05:20:44.929697118 +0000 UTC m=+18.039435263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs") pod "network-metrics-daemon-hqmnw" (UID: "ac746c00-7462-4686-9f96-7d5be8804348") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:36.929962 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:36.929652 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:36.929962 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:36.929814 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret podName:a540b042-b24e-416e-b30a-84346d4dd47f nodeName:}" failed. No retries permitted until 2026-09-11 05:20:44.929789847 +0000 UTC m=+18.039528008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret") pod "global-pull-secret-syncer-5fkvd" (UID: "a540b042-b24e-416e-b30a-84346d4dd47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:37.131057 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:37.131000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:37.131223 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:37.131196 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 05:20:37.131290 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:37.131226 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 05:20:37.131290 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:37.131239 2577 projected.go:196] Error preparing data for projected volume kube-api-access-2hrsf for pod openshift-network-diagnostics/network-check-target-g8zdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:37.131395 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:37.131303 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf podName:e699639b-b57e-4d9a-93b9-ec9b71a507fc nodeName:}" failed. No retries permitted until 2026-09-11 05:20:45.131285517 +0000 UTC m=+18.241023662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hrsf" (UniqueName: "kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf") pod "network-check-target-g8zdr" (UID: "e699639b-b57e-4d9a-93b9-ec9b71a507fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:37.351344 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:37.350683 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:37.351344 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:37.350789 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:37.351344 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:37.351152 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:37.351344 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:37.351253 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:38.349590 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:38.349539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:38.350007 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:38.349690 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:39.350414 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:39.350383 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:39.350902 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:39.350386 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:39.350902 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:39.350564 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:39.350902 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:39.350672 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:40.350187 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:40.350157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:40.350321 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:40.350269 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:41.350062 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:41.350035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:41.350460 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:41.350036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:41.350460 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:41.350132 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:41.350460 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:41.350225 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:42.350041 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:42.350011 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:42.350201 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:42.350115 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:43.350252 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:43.350219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:43.350689 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:43.350257 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:43.350689 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:43.350370 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:43.350689 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:43.350506 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:44.350109 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:44.350080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:44.350363 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:44.350197 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:44.991399 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:44.991366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:44.991534 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:44.991444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:44.991593 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:44.991532 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:44.991593 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:44.991560 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:44.991665 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:44.991619 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs podName:ac746c00-7462-4686-9f96-7d5be8804348 nodeName:}" failed. No retries permitted until 2026-09-11 05:21:00.99159912 +0000 UTC m=+34.101337265 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs") pod "network-metrics-daemon-hqmnw" (UID: "ac746c00-7462-4686-9f96-7d5be8804348") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:20:44.991665 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:44.991637 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret podName:a540b042-b24e-416e-b30a-84346d4dd47f nodeName:}" failed. No retries permitted until 2026-09-11 05:21:00.991628466 +0000 UTC m=+34.101366616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret") pod "global-pull-secret-syncer-5fkvd" (UID: "a540b042-b24e-416e-b30a-84346d4dd47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 05:20:45.192587 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:45.192503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:45.192718 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:45.192645 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 05:20:45.192718 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:45.192662 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 05:20:45.192718 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:45.192674 2577 projected.go:196] Error preparing data for projected volume kube-api-access-2hrsf for pod openshift-network-diagnostics/network-check-target-g8zdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:45.192860 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:45.192733 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf podName:e699639b-b57e-4d9a-93b9-ec9b71a507fc nodeName:}" failed. No retries permitted until 2026-09-11 05:21:01.192716872 +0000 UTC m=+34.302455017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hrsf" (UniqueName: "kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf") pod "network-check-target-g8zdr" (UID: "e699639b-b57e-4d9a-93b9-ec9b71a507fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:20:45.350234 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:45.350200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:45.350234 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:45.350228 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:45.350783 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:45.350329 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:45.350783 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:45.350455 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:46.350143 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:46.350112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:46.350308 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:46.350229 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:47.349982 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:47.349953 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:47.350412 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:47.350058 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:47.350412 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:47.350135 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:47.350412 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:47.350229 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:48.350000 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:48.349972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:48.350369 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:48.350086 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:49.350137 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.349717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:49.350619 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.349764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:49.350619 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:49.350260 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:49.350619 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:49.350292 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:49.482765 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.482703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6th6w" event={"ID":"830c602d-f84b-4211-8203-78d39e810f0a","Type":"ContainerStarted","Data":"0f430bdaef56c0538c6f87e67832493118ee97aa0a508135675a92d53df81f43"} Sep 11 05:20:49.484155 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.484133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" event={"ID":"05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95","Type":"ContainerStarted","Data":"d6e6d136ff4dfa7a796dd222f80ccb2637820d9c8b5f6f7b1e6251445b579705"} Sep 11 05:20:49.485646 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.485622 2577 generic.go:356] "Generic (PLEG): container finished" podID="2010dca1-86fb-471b-8a76-b9ff03a30d37" containerID="3071a234ddd1ea3a0a541a877cbaef56769dd3c4f2771b7b769f901a1d646f1b" exitCode=0 Sep 11 05:20:49.485731 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.485708 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2svzc" event={"ID":"2010dca1-86fb-471b-8a76-b9ff03a30d37","Type":"ContainerDied","Data":"3071a234ddd1ea3a0a541a877cbaef56769dd3c4f2771b7b769f901a1d646f1b"} Sep 11 05:20:49.487246 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.487220 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lk5n4" event={"ID":"3fa8974b-7bf1-4d99-884d-d5e923f8e568","Type":"ContainerStarted","Data":"447c0b8ebd26575d0850785d6152fe6952136df9120e6eb1673769ade2ae89f5"} Sep 11 05:20:49.488834 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.488815 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" event={"ID":"130e718b-93e4-4541-9a2a-9ee22ac205de","Type":"ContainerStarted","Data":"c8f9a7f6b6c48d2e0a121e2d689d88e8654b83da3ddbfcf6a89c5ee6b217ad28"} Sep 11 05:20:49.492501 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.490646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" event={"ID":"bee55f8a1340335ec8a8808d97d355d4","Type":"ContainerStarted","Data":"1ad5b2eb3858a0f04447e22aab9ca95f061ff61a6358c99b2d3a9a944d3eaec9"} Sep 11 05:20:49.495796 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.495772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" event={"ID":"f8b8a658-19bf-4a12-9a6a-e348e106b6d3","Type":"ContainerStarted","Data":"c5969b90da57714fb2ea9825a79b6b3d79ba7107ca998097a82458c713e0ca99"} Sep 11 05:20:49.495886 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.495870 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" event={"ID":"f8b8a658-19bf-4a12-9a6a-e348e106b6d3","Type":"ContainerStarted","Data":"fe8205df75f7a665d057bf70fa346af916a289f5b3977fd7445b09addc9caf90"} Sep 11 05:20:49.495943 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.495917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" event={"ID":"f8b8a658-19bf-4a12-9a6a-e348e106b6d3","Type":"ContainerStarted","Data":"bd3e532f55ba539f1fbb4e36179686b45904dc232ad239af950d2a83bf49ad2a"} Sep 11 05:20:49.495999 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.495959 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" event={"ID":"f8b8a658-19bf-4a12-9a6a-e348e106b6d3","Type":"ContainerStarted","Data":"43f396bb576ee93bd8c7e23d777183eefda6f02b2008643ec703bfdc6a49df10"} Sep 11 05:20:49.496050 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.496002 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" event={"ID":"f8b8a658-19bf-4a12-9a6a-e348e106b6d3","Type":"ContainerStarted","Data":"318b049a8eda317c4b2deda9e883110f0cec17842997bff980ac9ecc4600180b"} Sep 11 05:20:49.497321 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.497304 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvntc" event={"ID":"d4d6f060-e767-49e2-8c41-a32d0b1a83ae","Type":"ContainerStarted","Data":"a7e821ace1656403879304e6479e1d38a2a6e09b28b4c9563b9bb3f174a85d99"} Sep 11 05:20:49.498748 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.498730 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gvw7p" event={"ID":"4ad2bdd1-8798-4983-9878-0060b20d0e3e","Type":"ContainerStarted","Data":"68ffbc25848f2c57bdf70acfc101991bdda45f42c13c4370150c5e618713b7a0"} Sep 11 05:20:49.500531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.500495 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6th6w" podStartSLOduration=2.346285543 podStartE2EDuration="21.500483997s" totalImagesPullingTime="19.154198454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:49.500311356 +0000 UTC m=+22.610049552" watchObservedRunningTime="2026-09-11 05:20:49.500483997 +0000 UTC m=+22.610222163" Sep 11 05:20:49.513957 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.513928 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mvntc" podStartSLOduration=2.443441691 podStartE2EDuration="21.513917418s" totalImagesPullingTime="19.070475727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:49.513607817 +0000 UTC m=+22.623345982" watchObservedRunningTime="2026-09-11 05:20:49.513917418 +0000 UTC m=+22.623655583" Sep 11 05:20:49.563016 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.562967 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" podStartSLOduration=20.562948276 podStartE2EDuration="20.562948276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:49.562671953 +0000 UTC m=+22.672410120" watchObservedRunningTime="2026-09-11 05:20:49.562948276 +0000 UTC m=+22.672686445" Sep 11 05:20:49.563396 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.563371 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lk5n4" podStartSLOduration=6.330113575 podStartE2EDuration="21.563364417s" totalImagesPullingTime="15.233250842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:49.547095493 +0000 UTC m=+22.656833658" watchObservedRunningTime="2026-09-11 05:20:49.563364417 +0000 UTC m=+22.673102583" Sep 11 05:20:49.578697 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:49.578636 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fzhcj" podStartSLOduration=2.502798855 podStartE2EDuration="21.578619019s" totalImagesPullingTime="19.075820164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:49.57836309 +0000 UTC m=+22.688101258" watchObservedRunningTime="2026-09-11 05:20:49.578619019 +0000 UTC m=+22.688357185" Sep 11 05:20:50.262915 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.262772 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 05:20:50.300863 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.300774 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T05:20:50.262913934Z","UUID":"79de7f4a-5af2-4fe5-a3ec-dd41fe716078","Handler":null,"Name":"","Endpoint":""} Sep 11 05:20:50.301998 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.301983 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 05:20:50.302072 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.302009 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 05:20:50.349779 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.349756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:50.349869 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:50.349842 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:50.503873 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.503841 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" event={"ID":"f8b8a658-19bf-4a12-9a6a-e348e106b6d3","Type":"ContainerStarted","Data":"768578c132a602bff8b822bd3fa4438a083b7be816d7d4c60d7c4a9dc5eb42fc"} Sep 11 05:20:50.505328 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.505306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m7b82" event={"ID":"d1f7ce08-a06a-4063-92e7-8f2c02044ba8","Type":"ContainerStarted","Data":"133fe31c0c6b4dd09e8acbd32318d6600dd199603dd475eb5ad2fbab8313c5fa"} Sep 11 05:20:50.507155 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.507133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" event={"ID":"130e718b-93e4-4541-9a2a-9ee22ac205de","Type":"ContainerStarted","Data":"a29e93e2253e4894ccf8f30b370b54a16fb026c4ab50386a0daa1fbf247f8e27"} Sep 11 05:20:50.520828 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.520790 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gvw7p" podStartSLOduration=7.292372474 podStartE2EDuration="22.520777932s" totalImagesPullingTime="15.228405458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:49.591609363 +0000 UTC m=+22.701347530" watchObservedRunningTime="2026-09-11 05:20:50.520777932 +0000 UTC m=+23.630516097" Sep 11 05:20:50.521115 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:50.521086 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m7b82" podStartSLOduration=3.422552877 podStartE2EDuration="22.521080536s" totalImagesPullingTime="19.098527659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:50.520493688 +0000 UTC m=+23.630231855" watchObservedRunningTime="2026-09-11 05:20:50.521080536 +0000 UTC m=+23.630818703" Sep 11 05:20:51.350290 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:51.350204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:51.350290 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:51.350250 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:51.350464 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:51.350345 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:51.350464 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:51.350452 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:51.511844 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:51.511797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" event={"ID":"130e718b-93e4-4541-9a2a-9ee22ac205de","Type":"ContainerStarted","Data":"c41829e26078c6574fb30f96a10a005428ad2b1457c5fe791f8d2388fc53af55"} Sep 11 05:20:51.533490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:51.533446 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bp9k" podStartSLOduration=2.477111368 podStartE2EDuration="23.533432747s" totalImagesPullingTime="21.056321379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:51.533230808 +0000 UTC m=+24.642968974" watchObservedRunningTime="2026-09-11 05:20:51.533432747 +0000 UTC m=+24.643170912" Sep 11 05:20:52.350098 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:52.350068 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:52.350283 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:52.350175 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:52.517830 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:52.517782 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" event={"ID":"f8b8a658-19bf-4a12-9a6a-e348e106b6d3","Type":"ContainerStarted","Data":"b6ddb63e4d6e28cc93fc99e51d7d1158256a7939f0d733753d3155ab721e2265"} Sep 11 05:20:53.349434 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:53.349404 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:53.349713 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:53.349668 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:53.349805 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:53.349724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:53.349856 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:53.349831 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:53.945527 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:53.945239 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:53.946061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:53.946042 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:54.349510 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:54.349477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:54.349731 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:54.349591 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:54.527068 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:54.527015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" event={"ID":"f8b8a658-19bf-4a12-9a6a-e348e106b6d3","Type":"ContainerStarted","Data":"d769e5294a46c29d60ab7441f0f1fb33e3df92ebd063929d3b31b83dab9ef9dd"} Sep 11 05:20:54.553203 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:54.553154 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" podStartSLOduration=7.454449307 podStartE2EDuration="26.553140576s" totalImagesPullingTime="19.098691269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:20:54.551746823 +0000 UTC m=+27.661484989" watchObservedRunningTime="2026-09-11 05:20:54.553140576 +0000 UTC m=+27.662878742" Sep 11 05:20:54.645118 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:54.645034 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:55.349923 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.349870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:55.350564 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.349872 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:55.350564 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:55.350012 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:55.350564 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:55.350072 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:55.529474 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.529441 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:55.529633 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.529533 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:55.544576 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.544536 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:55.544710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.544652 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:20:55.720724 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.720606 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5fkvd"] Sep 11 05:20:55.720875 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.720746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:55.720875 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:55.720855 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:55.723858 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.723829 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8zdr"] Sep 11 05:20:55.724001 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.723923 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:55.724051 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:55.724021 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:55.724543 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.724516 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hqmnw"] Sep 11 05:20:55.724664 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:55.724632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:55.724757 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:55.724736 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:56.533340 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:56.533160 2577 generic.go:356] "Generic (PLEG): container finished" podID="2010dca1-86fb-471b-8a76-b9ff03a30d37" containerID="395cf6e57e49581083ffbe1b7a2e5bf824eafb66a871cfe38ed039d0ef249153" exitCode=0 Sep 11 05:20:56.533924 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:56.533240 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2svzc" event={"ID":"2010dca1-86fb-471b-8a76-b9ff03a30d37","Type":"ContainerDied","Data":"395cf6e57e49581083ffbe1b7a2e5bf824eafb66a871cfe38ed039d0ef249153"} Sep 11 05:20:57.275760 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:57.275736 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lk5n4_3fa8974b-7bf1-4d99-884d-d5e923f8e568/dns-node-resolver/0.log" Sep 11 05:20:57.351324 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:57.351295 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:57.351490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:57.351402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:57.351490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:57.351449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:57.351490 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:57.351445 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:57.351623 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:57.351527 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:57.351663 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:57.351643 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:20:58.048254 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:58.048227 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvntc_d4d6f060-e767-49e2-8c41-a32d0b1a83ae/node-ca/0.log" Sep 11 05:20:58.442850 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:58.442770 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:58.443007 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:58.442951 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 05:20:58.443363 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:58.443342 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gvw7p" Sep 11 05:20:58.539675 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:58.539646 2577 generic.go:356] "Generic (PLEG): container finished" podID="2010dca1-86fb-471b-8a76-b9ff03a30d37" containerID="c8a7a93243a55664685cb4c1e76d496826e30ebde62d1643eb9ef981a3bc0a56" exitCode=0 Sep 11 05:20:58.539877 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:58.539728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2svzc" event={"ID":"2010dca1-86fb-471b-8a76-b9ff03a30d37","Type":"ContainerDied","Data":"c8a7a93243a55664685cb4c1e76d496826e30ebde62d1643eb9ef981a3bc0a56"} Sep 11 05:20:59.349827 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:59.349792 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:20:59.350363 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:59.349792 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:20:59.350363 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:59.349898 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:20:59.350363 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:20:59.349792 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:20:59.350363 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:59.349970 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:20:59.350363 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:20:59.350053 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:00.545745 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:00.545713 2577 generic.go:356] "Generic (PLEG): container finished" podID="2010dca1-86fb-471b-8a76-b9ff03a30d37" containerID="367dc1a9e3bd14fc47454b4f20dc504dbf61bbc049a60f543d379f79f30c7ecb" exitCode=0 Sep 11 05:21:00.546230 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:00.545772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2svzc" event={"ID":"2010dca1-86fb-471b-8a76-b9ff03a30d37","Type":"ContainerDied","Data":"367dc1a9e3bd14fc47454b4f20dc504dbf61bbc049a60f543d379f79f30c7ecb"} Sep 11 05:21:01.004497 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:01.004465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:01.004686 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:01.004513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:01.004686 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.004627 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:21:01.004686 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.004684 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs podName:ac746c00-7462-4686-9f96-7d5be8804348 nodeName:}" failed. No retries permitted until 2026-09-11 05:21:33.004667731 +0000 UTC m=+66.114405881 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs") pod "network-metrics-daemon-hqmnw" (UID: "ac746c00-7462-4686-9f96-7d5be8804348") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 05:21:01.004890 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.004627 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 05:21:01.004890 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.004792 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret podName:a540b042-b24e-416e-b30a-84346d4dd47f nodeName:}" failed. No retries permitted until 2026-09-11 05:21:33.004779647 +0000 UTC m=+66.114517791 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret") pod "global-pull-secret-syncer-5fkvd" (UID: "a540b042-b24e-416e-b30a-84346d4dd47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 05:21:01.205959 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:01.205922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:01.206149 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.206106 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 05:21:01.206149 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.206133 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 05:21:01.206149 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.206148 2577 projected.go:196] Error preparing data for projected volume kube-api-access-2hrsf for pod openshift-network-diagnostics/network-check-target-g8zdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:21:01.206279 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.206213 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf podName:e699639b-b57e-4d9a-93b9-ec9b71a507fc nodeName:}" failed. No retries permitted until 2026-09-11 05:21:33.206195247 +0000 UTC m=+66.315933393 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hrsf" (UniqueName: "kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf") pod "network-check-target-g8zdr" (UID: "e699639b-b57e-4d9a-93b9-ec9b71a507fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 05:21:01.349981 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:01.349903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:01.350136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:01.349904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:01.350136 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.350030 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:01.350136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:01.349906 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:01.350136 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.350092 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:01.350347 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:01.350160 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:03.350048 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:03.350014 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:03.350048 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:03.350046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:03.350441 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:03.350014 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:03.350441 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:03.350134 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:03.350441 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:03.350197 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:03.350441 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:03.350293 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:05.350162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:05.349963 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:05.350606 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:05.350016 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:05.350606 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:05.350248 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:05.350606 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:05.350028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:05.350606 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:05.350304 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:05.350606 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:05.350419 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:05.560625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:05.560597 2577 generic.go:356] "Generic (PLEG): container finished" podID="2010dca1-86fb-471b-8a76-b9ff03a30d37" containerID="e1abe5507123670edee3f34926aca8c693a7a6d6718ffcd00800de115d585da2" exitCode=0 Sep 11 05:21:05.560797 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:05.560638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2svzc" event={"ID":"2010dca1-86fb-471b-8a76-b9ff03a30d37","Type":"ContainerDied","Data":"e1abe5507123670edee3f34926aca8c693a7a6d6718ffcd00800de115d585da2"} Sep 11 05:21:06.566702 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:06.566667 2577 generic.go:356] "Generic (PLEG): container finished" podID="2010dca1-86fb-471b-8a76-b9ff03a30d37" containerID="8d85d993049681dfe983122dfb2be5216ec8e1175ee2c6de1d5f6f1cae7bcb9f" exitCode=0 Sep 11 05:21:06.567178 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:06.566717 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2svzc" event={"ID":"2010dca1-86fb-471b-8a76-b9ff03a30d37","Type":"ContainerDied","Data":"8d85d993049681dfe983122dfb2be5216ec8e1175ee2c6de1d5f6f1cae7bcb9f"} Sep 11 05:21:07.350919 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:07.350889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:07.351077 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:07.350984 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:07.351077 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:07.351070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:07.351176 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:07.351151 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:07.351211 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:07.351178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:07.351268 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:07.351252 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:07.571919 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:07.571888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2svzc" event={"ID":"2010dca1-86fb-471b-8a76-b9ff03a30d37","Type":"ContainerStarted","Data":"bcb23f3ae4a51b387e8341c59dbccc99983ba1df179907444ec2f6bd0fd5e2a9"} Sep 11 05:21:09.349465 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:09.349430 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:09.349834 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:09.349435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:09.349834 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:09.349530 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:09.349834 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:09.349435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:09.349834 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:09.349622 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:09.349834 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:09.349712 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:11.349909 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:11.349870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:11.349909 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:11.349900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:11.350400 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:11.349879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:11.350400 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:11.349968 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:11.350400 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:11.350101 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:11.350400 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:11.350130 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:13.349885 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:13.349852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:13.350345 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:13.349852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:13.350345 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:13.349862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:13.350345 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:13.349945 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:13.350345 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:13.350152 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:13.350345 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:13.350079 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:15.349499 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:15.349461 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:15.349892 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:15.349461 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:15.349892 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:15.349465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:15.349892 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:15.349629 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:15.349892 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:15.349729 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:15.349892 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:15.349808 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:17.350886 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:17.350852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:17.351337 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:17.350943 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:17.351337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:17.351031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:17.351337 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:17.351142 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:17.351337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:17.351217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:17.351337 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:17.351296 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:19.349633 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:19.349600 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:19.349633 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:19.349623 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:19.350095 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:19.349603 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:19.350095 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:19.349725 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqmnw" podUID="ac746c00-7462-4686-9f96-7d5be8804348" Sep 11 05:21:19.350330 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:19.350195 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8zdr" podUID="e699639b-b57e-4d9a-93b9-ec9b71a507fc" Sep 11 05:21:19.350389 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:19.350338 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5fkvd" podUID="a540b042-b24e-416e-b30a-84346d4dd47f" Sep 11 05:21:20.653847 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.653770 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeReady" Sep 11 05:21:20.654183 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.653904 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 05:21:20.745102 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.745046 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2svzc" podStartSLOduration=20.776208255 podStartE2EDuration="52.745029857s" totalImagesPullingTime="31.968821602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:07.591908567 +0000 UTC m=+40.701646746" watchObservedRunningTime="2026-09-11 05:21:20.745029857 +0000 UTC m=+53.854768022" Sep 11 05:21:20.745260 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.745227 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kw5tg"] Sep 11 05:21:20.766926 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.766900 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jbzzm"] Sep 11 05:21:20.767083 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.767028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:20.769699 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.769671 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7z8t4\"" Sep 11 05:21:20.770166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.770129 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 05:21:20.770421 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.770402 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 05:21:20.790974 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.790947 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kw5tg"] Sep 11 05:21:20.791099 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.790982 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jbzzm"] Sep 11 05:21:20.791099 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.791004 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vch8h"] Sep 11 05:21:20.791099 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.791074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jbzzm" Sep 11 05:21:20.794055 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.794032 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 05:21:20.794198 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.794181 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 05:21:20.794318 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.794298 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-skmld\"" Sep 11 05:21:20.794423 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.794403 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 05:21:20.815021 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.814988 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vch8h"] Sep 11 05:21:20.815157 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.815098 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:20.817632 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.817610 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 05:21:20.817738 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.817638 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 05:21:20.817879 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.817865 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 05:21:20.817943 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.817866 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rpvwz\"" Sep 11 05:21:20.817998 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.817976 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 05:21:20.951162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951062 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-tmp-dir\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:20.951162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dab642a-b903-4531-a22a-c796d043aaac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:20.951162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-metrics-tls\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:20.951162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7tr\" (UniqueName: \"kubernetes.io/projected/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-kube-api-access-6h7tr\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:20.951405 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-config-volume\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:20.951405 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8dg\" (UniqueName: \"kubernetes.io/projected/4dab642a-b903-4531-a22a-c796d043aaac-kube-api-access-lh8dg\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:20.951405 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7232efa1-0a8b-47fe-a41a-55f84611ff7e-cert\") pod \"ingress-canary-jbzzm\" (UID: \"7232efa1-0a8b-47fe-a41a-55f84611ff7e\") " pod="openshift-ingress-canary/ingress-canary-jbzzm" Sep 11 05:21:20.951405 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dab642a-b903-4531-a22a-c796d043aaac-data-volume\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:20.951405 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dab642a-b903-4531-a22a-c796d043aaac-crio-socket\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:20.951405 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4w48\" (UniqueName: \"kubernetes.io/projected/7232efa1-0a8b-47fe-a41a-55f84611ff7e-kube-api-access-w4w48\") pod \"ingress-canary-jbzzm\" (UID: \"7232efa1-0a8b-47fe-a41a-55f84611ff7e\") " pod="openshift-ingress-canary/ingress-canary-jbzzm" Sep 11 05:21:20.951614 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:20.951441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dab642a-b903-4531-a22a-c796d043aaac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.037190 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.037152 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx"] Sep 11 05:21:21.052261 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dab642a-b903-4531-a22a-c796d043aaac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.052427 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-tmp-dir\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:21.052427 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dab642a-b903-4531-a22a-c796d043aaac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.052427 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-metrics-tls\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:21.052427 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7tr\" (UniqueName: \"kubernetes.io/projected/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-kube-api-access-6h7tr\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:21.052427 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-config-volume\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:21.052427 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8dg\" (UniqueName: \"kubernetes.io/projected/4dab642a-b903-4531-a22a-c796d043aaac-kube-api-access-lh8dg\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.052736 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7232efa1-0a8b-47fe-a41a-55f84611ff7e-cert\") pod \"ingress-canary-jbzzm\" (UID: \"7232efa1-0a8b-47fe-a41a-55f84611ff7e\") " pod="openshift-ingress-canary/ingress-canary-jbzzm" Sep 11 05:21:21.052736 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dab642a-b903-4531-a22a-c796d043aaac-data-volume\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.052736 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dab642a-b903-4531-a22a-c796d043aaac-crio-socket\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.052736 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4w48\" (UniqueName: \"kubernetes.io/projected/7232efa1-0a8b-47fe-a41a-55f84611ff7e-kube-api-access-w4w48\") pod \"ingress-canary-jbzzm\" (UID: \"7232efa1-0a8b-47fe-a41a-55f84611ff7e\") " pod="openshift-ingress-canary/ingress-canary-jbzzm" Sep 11 05:21:21.052922 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.052840 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dab642a-b903-4531-a22a-c796d043aaac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.053254 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.053208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dab642a-b903-4531-a22a-c796d043aaac-crio-socket\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.053372 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.053307 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-tmp-dir\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:21.053425 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.053382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dab642a-b903-4531-a22a-c796d043aaac-data-volume\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.053460 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.053440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-config-volume\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:21.056810 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.056786 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-metrics-tls\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:21.056907 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.056826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dab642a-b903-4531-a22a-c796d043aaac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.056907 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.056856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7232efa1-0a8b-47fe-a41a-55f84611ff7e-cert\") pod \"ingress-canary-jbzzm\" (UID: \"7232efa1-0a8b-47fe-a41a-55f84611ff7e\") " pod="openshift-ingress-canary/ingress-canary-jbzzm" Sep 11 05:21:21.057239 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.057222 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx"] Sep 11 05:21:21.057326 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.057314 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" Sep 11 05:21:21.059744 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.059729 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7szst\"" Sep 11 05:21:21.060062 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.060048 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 05:21:21.063956 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.063928 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7tr\" (UniqueName: \"kubernetes.io/projected/f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0-kube-api-access-6h7tr\") pod \"dns-default-kw5tg\" (UID: \"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0\") " pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:21.064796 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.064778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8dg\" (UniqueName: \"kubernetes.io/projected/4dab642a-b903-4531-a22a-c796d043aaac-kube-api-access-lh8dg\") pod \"insights-runtime-extractor-vch8h\" (UID: \"4dab642a-b903-4531-a22a-c796d043aaac\") " pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.065082 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.065058 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4w48\" (UniqueName: \"kubernetes.io/projected/7232efa1-0a8b-47fe-a41a-55f84611ff7e-kube-api-access-w4w48\") pod \"ingress-canary-jbzzm\" (UID: \"7232efa1-0a8b-47fe-a41a-55f84611ff7e\") " pod="openshift-ingress-canary/ingress-canary-jbzzm" Sep 11 05:21:21.075748 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.075726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:21.100158 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.100126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jbzzm" Sep 11 05:21:21.123831 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.123804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vch8h" Sep 11 05:21:21.153670 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.153635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/64e7d3d0-8e1e-4ec6-9755-d8313f2da409-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jpwsx\" (UID: \"64e7d3d0-8e1e-4ec6-9755-d8313f2da409\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" Sep 11 05:21:21.254269 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.254223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/64e7d3d0-8e1e-4ec6-9755-d8313f2da409-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jpwsx\" (UID: \"64e7d3d0-8e1e-4ec6-9755-d8313f2da409\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" Sep 11 05:21:21.254414 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:21.254358 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 11 05:21:21.254477 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:21.254416 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e7d3d0-8e1e-4ec6-9755-d8313f2da409-tls-certificates podName:64e7d3d0-8e1e-4ec6-9755-d8313f2da409 nodeName:}" failed. No retries permitted until 2026-09-11 05:21:21.754399496 +0000 UTC m=+54.864137641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/64e7d3d0-8e1e-4ec6-9755-d8313f2da409-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-jpwsx" (UID: "64e7d3d0-8e1e-4ec6-9755-d8313f2da409") : secret "prometheus-operator-admission-webhook-tls" not found Sep 11 05:21:21.349963 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.349925 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:21.350131 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.349932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:21.350131 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.349932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:21.352731 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.352686 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 05:21:21.352868 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.352783 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 05:21:21.352868 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.352783 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 05:21:21.352984 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.352891 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dpzsh\"" Sep 11 05:21:21.353166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.353149 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g92vd\"" Sep 11 05:21:21.353217 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.353169 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 05:21:21.400400 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.400364 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jbzzm"] Sep 11 05:21:21.403529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.403501 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kw5tg"] Sep 11 05:21:21.404775 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:21.404739 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7232efa1_0a8b_47fe_a41a_55f84611ff7e.slice/crio-bff36736c6fd054a27d0b854c98aaec29f84f70eef3cfe1d01b2b4efc4f4d07f WatchSource:0}: Error finding container bff36736c6fd054a27d0b854c98aaec29f84f70eef3cfe1d01b2b4efc4f4d07f: Status 404 returned error can't find the container with id bff36736c6fd054a27d0b854c98aaec29f84f70eef3cfe1d01b2b4efc4f4d07f Sep 11 05:21:21.407242 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:21.407220 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b7e3b3_9dbb_4f5a_8476_60ee7e55aee0.slice/crio-19711fd0572eb0c35d96880ca2b2c4b6ede307476446f229f93b7dbf1243855a WatchSource:0}: Error finding container 19711fd0572eb0c35d96880ca2b2c4b6ede307476446f229f93b7dbf1243855a: Status 404 returned error can't find the container with id 19711fd0572eb0c35d96880ca2b2c4b6ede307476446f229f93b7dbf1243855a Sep 11 05:21:21.436466 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.436426 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vch8h"] Sep 11 05:21:21.439823 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:21.439795 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dab642a_b903_4531_a22a_c796d043aaac.slice/crio-f8cf53d985ecdcb467084fdd768bbad596cd490cb4ebb2811e03b7be545ae8ee WatchSource:0}: Error finding container f8cf53d985ecdcb467084fdd768bbad596cd490cb4ebb2811e03b7be545ae8ee: Status 404 returned error can't find the container with id f8cf53d985ecdcb467084fdd768bbad596cd490cb4ebb2811e03b7be545ae8ee Sep 11 05:21:21.604319 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.604280 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jbzzm" event={"ID":"7232efa1-0a8b-47fe-a41a-55f84611ff7e","Type":"ContainerStarted","Data":"bff36736c6fd054a27d0b854c98aaec29f84f70eef3cfe1d01b2b4efc4f4d07f"} Sep 11 05:21:21.605890 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.605859 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vch8h" event={"ID":"4dab642a-b903-4531-a22a-c796d043aaac","Type":"ContainerStarted","Data":"70e8a8bffdbe04b5858cad9c18d93ffe5b632847680b799028c6161c810dbfc8"} Sep 11 05:21:21.606000 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.605922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vch8h" event={"ID":"4dab642a-b903-4531-a22a-c796d043aaac","Type":"ContainerStarted","Data":"f8cf53d985ecdcb467084fdd768bbad596cd490cb4ebb2811e03b7be545ae8ee"} Sep 11 05:21:21.606955 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.606933 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kw5tg" event={"ID":"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0","Type":"ContainerStarted","Data":"19711fd0572eb0c35d96880ca2b2c4b6ede307476446f229f93b7dbf1243855a"} Sep 11 05:21:21.758543 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.758510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/64e7d3d0-8e1e-4ec6-9755-d8313f2da409-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jpwsx\" (UID: \"64e7d3d0-8e1e-4ec6-9755-d8313f2da409\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" Sep 11 05:21:21.760864 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.760844 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/64e7d3d0-8e1e-4ec6-9755-d8313f2da409-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jpwsx\" (UID: \"64e7d3d0-8e1e-4ec6-9755-d8313f2da409\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" Sep 11 05:21:21.977717 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:21.977679 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" Sep 11 05:21:22.254266 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:22.254235 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx"] Sep 11 05:21:22.258020 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:22.257987 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e7d3d0_8e1e_4ec6_9755_d8313f2da409.slice/crio-0bf474acca3726f32b21451b71e12db68f023da11aeeedb649110e75deb993ea WatchSource:0}: Error finding container 0bf474acca3726f32b21451b71e12db68f023da11aeeedb649110e75deb993ea: Status 404 returned error can't find the container with id 0bf474acca3726f32b21451b71e12db68f023da11aeeedb649110e75deb993ea Sep 11 05:21:22.610777 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:22.610722 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" event={"ID":"64e7d3d0-8e1e-4ec6-9755-d8313f2da409","Type":"ContainerStarted","Data":"0bf474acca3726f32b21451b71e12db68f023da11aeeedb649110e75deb993ea"} Sep 11 05:21:24.617514 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.617426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jbzzm" event={"ID":"7232efa1-0a8b-47fe-a41a-55f84611ff7e","Type":"ContainerStarted","Data":"bccfc9a8478be03b5c9c39c9cb1ec5249e39969a6834f16043642b8af8642018"} Sep 11 05:21:24.619318 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.619285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kw5tg" event={"ID":"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0","Type":"ContainerStarted","Data":"086d85cd09ea1b9dfd2b1614e46417bcd5d71ebf297fa9ae14aa7298917655fa"} Sep 11 05:21:24.619456 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.619341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kw5tg" event={"ID":"f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0","Type":"ContainerStarted","Data":"607ae1bde2eab806a518e4c2f7a9bbd7739e80d8f8089006d0b73a85942a8f0c"} Sep 11 05:21:24.619456 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.619372 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:24.620828 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.620802 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" event={"ID":"64e7d3d0-8e1e-4ec6-9755-d8313f2da409","Type":"ContainerStarted","Data":"d77cb4ed5a24991fdf099568abcad3413bda53c481c1a00601d851b2dfc1b0a5"} Sep 11 05:21:24.621034 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.621013 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" Sep 11 05:21:24.625684 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.625664 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" Sep 11 05:21:24.633764 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.633721 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jbzzm" podStartSLOduration=1.685709282 podStartE2EDuration="4.633706628s" totalImagesPullingTime="2.947997346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:24.633150058 +0000 UTC m=+57.742888225" watchObservedRunningTime="2026-09-11 05:21:24.633706628 +0000 UTC m=+57.743444824" Sep 11 05:21:24.652641 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.652545 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kw5tg" podStartSLOduration=1.7129942649999998 podStartE2EDuration="4.652524627s" totalImagesPullingTime="2.939530362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:24.652321872 +0000 UTC m=+57.762060039" watchObservedRunningTime="2026-09-11 05:21:24.652524627 +0000 UTC m=+57.762262794" Sep 11 05:21:24.666585 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:24.666439 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jpwsx" podStartSLOduration=1.57656946 podStartE2EDuration="3.666417088s" totalImagesPullingTime="2.089847628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:24.666216873 +0000 UTC m=+57.775955039" watchObservedRunningTime="2026-09-11 05:21:24.666417088 +0000 UTC m=+57.776155261" Sep 11 05:21:25.109067 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.109032 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-44bvn"] Sep 11 05:21:25.110960 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.110945 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.113443 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.113422 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 05:21:25.113443 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.113436 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 05:21:25.113639 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.113619 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 05:21:25.113739 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.113723 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 05:21:25.113800 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.113783 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-drz4b\"" Sep 11 05:21:25.114623 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.114607 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 05:21:25.121972 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.121952 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-44bvn"] Sep 11 05:21:25.282749 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.282712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2116f63-04d2-4c2c-9882-878e5079dc7c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.282914 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.282768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2116f63-04d2-4c2c-9882-878e5079dc7c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.282914 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.282834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2116f63-04d2-4c2c-9882-878e5079dc7c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.282914 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.282864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snx5k\" (UniqueName: \"kubernetes.io/projected/a2116f63-04d2-4c2c-9882-878e5079dc7c-kube-api-access-snx5k\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.383918 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.383819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2116f63-04d2-4c2c-9882-878e5079dc7c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.383918 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.383909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2116f63-04d2-4c2c-9882-878e5079dc7c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.384144 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.384055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snx5k\" (UniqueName: \"kubernetes.io/projected/a2116f63-04d2-4c2c-9882-878e5079dc7c-kube-api-access-snx5k\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.384144 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.384090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2116f63-04d2-4c2c-9882-878e5079dc7c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.384256 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:25.384182 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 11 05:21:25.384256 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:25.384251 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2116f63-04d2-4c2c-9882-878e5079dc7c-prometheus-operator-tls podName:a2116f63-04d2-4c2c-9882-878e5079dc7c nodeName:}" failed. No retries permitted until 2026-09-11 05:21:25.884230741 +0000 UTC m=+58.993968886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a2116f63-04d2-4c2c-9882-878e5079dc7c-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-44bvn" (UID: "a2116f63-04d2-4c2c-9882-878e5079dc7c") : secret "prometheus-operator-tls" not found Sep 11 05:21:25.384690 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.384669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2116f63-04d2-4c2c-9882-878e5079dc7c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.386204 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.386183 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2116f63-04d2-4c2c-9882-878e5079dc7c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.393693 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.393672 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snx5k\" (UniqueName: \"kubernetes.io/projected/a2116f63-04d2-4c2c-9882-878e5079dc7c-kube-api-access-snx5k\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.628533 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.628495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vch8h" event={"ID":"4dab642a-b903-4531-a22a-c796d043aaac","Type":"ContainerStarted","Data":"63945c1a7058a0caba9061dd1ee83dd1c2286b22e66f6f7098df62c5ab1c5478"} Sep 11 05:21:25.888671 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.888634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2116f63-04d2-4c2c-9882-878e5079dc7c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:25.891429 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:25.891391 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2116f63-04d2-4c2c-9882-878e5079dc7c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-44bvn\" (UID: \"a2116f63-04d2-4c2c-9882-878e5079dc7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:26.019481 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.019438 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" Sep 11 05:21:26.294303 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.294267 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-44bvn"] Sep 11 05:21:26.317321 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.317290 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-597467cd95-qfpks"] Sep 11 05:21:26.319648 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.319630 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.323237 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.323021 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6lz56\"" Sep 11 05:21:26.323237 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.323101 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 05:21:26.323237 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.323116 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 05:21:26.323237 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.323100 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 05:21:26.323237 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.323102 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 05:21:26.323587 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.323251 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 05:21:26.323587 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.323475 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 05:21:26.323706 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.323683 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 05:21:26.331775 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.331746 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597467cd95-qfpks"] Sep 11 05:21:26.391634 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.391595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-console-config\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.391634 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.391636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-oauth-serving-cert\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.391876 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.391691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwlw5\" (UniqueName: \"kubernetes.io/projected/b2758633-5ec4-4b49-91a9-5db89f072899-kube-api-access-rwlw5\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.391876 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.391737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-serving-cert\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.391876 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.391761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-service-ca\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.392004 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.391871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-oauth-config\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.470217 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:26.470185 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2116f63_04d2_4c2c_9882_878e5079dc7c.slice/crio-e2f2eebb13faa738f012d507d07adfacd5aaa82379d62087179bbfcef1a7df5f WatchSource:0}: Error finding container e2f2eebb13faa738f012d507d07adfacd5aaa82379d62087179bbfcef1a7df5f: Status 404 returned error can't find the container with id e2f2eebb13faa738f012d507d07adfacd5aaa82379d62087179bbfcef1a7df5f Sep 11 05:21:26.492768 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.492746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-oauth-config\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.492865 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.492791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-console-config\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.493030 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.493006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-oauth-serving-cert\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.493107 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.493079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwlw5\" (UniqueName: \"kubernetes.io/projected/b2758633-5ec4-4b49-91a9-5db89f072899-kube-api-access-rwlw5\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.493170 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.493143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-serving-cert\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.493230 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.493169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-service-ca\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.493466 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.493447 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-console-config\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.493814 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.493723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-oauth-serving-cert\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.493927 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.493905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-service-ca\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.495367 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.495348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-oauth-config\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.495666 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.495647 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-serving-cert\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.503483 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.503461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwlw5\" (UniqueName: \"kubernetes.io/projected/b2758633-5ec4-4b49-91a9-5db89f072899-kube-api-access-rwlw5\") pod \"console-597467cd95-qfpks\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.630526 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.630429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:26.633656 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.633623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vch8h" event={"ID":"4dab642a-b903-4531-a22a-c796d043aaac","Type":"ContainerStarted","Data":"36b5223fbcf89e1869edb3793eee15f2882c47967c0ae0d192dc48ec3aa8dffd"} Sep 11 05:21:26.634989 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.634965 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" event={"ID":"a2116f63-04d2-4c2c-9882-878e5079dc7c","Type":"ContainerStarted","Data":"e2f2eebb13faa738f012d507d07adfacd5aaa82379d62087179bbfcef1a7df5f"} Sep 11 05:21:26.651232 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.651176 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vch8h" podStartSLOduration=1.83193191 podStartE2EDuration="6.651153299s" totalImagesPullingTime="4.819221389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:21:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:26.650218708 +0000 UTC m=+59.759956875" watchObservedRunningTime="2026-09-11 05:21:26.651153299 +0000 UTC m=+59.760891468" Sep 11 05:21:26.897677 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:26.897600 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597467cd95-qfpks"] Sep 11 05:21:26.900461 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:26.900435 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2758633_5ec4_4b49_91a9_5db89f072899.slice/crio-f4963f5c564a225aea808e0dc593f490917478cdecf6ffb728640ede6b6b4844 WatchSource:0}: Error finding container f4963f5c564a225aea808e0dc593f490917478cdecf6ffb728640ede6b6b4844: Status 404 returned error can't find the container with id f4963f5c564a225aea808e0dc593f490917478cdecf6ffb728640ede6b6b4844 Sep 11 05:21:27.551249 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:27.551221 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srxzk" Sep 11 05:21:27.641135 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:27.641088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597467cd95-qfpks" event={"ID":"b2758633-5ec4-4b49-91a9-5db89f072899","Type":"ContainerStarted","Data":"f4963f5c564a225aea808e0dc593f490917478cdecf6ffb728640ede6b6b4844"} Sep 11 05:21:28.646221 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:28.646176 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" event={"ID":"a2116f63-04d2-4c2c-9882-878e5079dc7c","Type":"ContainerStarted","Data":"158d89cc3e0c187428fad3a26e0cba7194a0f85ee6925d101248d38f42117c1e"} Sep 11 05:21:28.646707 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:28.646246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" event={"ID":"a2116f63-04d2-4c2c-9882-878e5079dc7c","Type":"ContainerStarted","Data":"6e01dea7b731ad8eb03c210aec74fe6ea35eb72cf37bfe6d1455b22eb79d8557"} Sep 11 05:21:28.666035 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:28.665979 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-44bvn" podStartSLOduration=1.945879269 podStartE2EDuration="3.665957417s" totalImagesPullingTime="1.720078148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:28.664220194 +0000 UTC m=+61.773958360" watchObservedRunningTime="2026-09-11 05:21:28.665957417 +0000 UTC m=+61.775695584" Sep 11 05:21:30.450220 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.450184 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8"] Sep 11 05:21:30.452337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.452319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.454648 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.454620 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 11 05:21:30.454795 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.454661 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5z4nt\"" Sep 11 05:21:30.454795 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.454693 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 11 05:21:30.470478 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.470437 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8"] Sep 11 05:21:30.478503 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.478474 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c"] Sep 11 05:21:30.481166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.481038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.484945 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.484479 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 05:21:30.484945 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.484818 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 05:21:30.484945 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.484838 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 05:21:30.485210 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.484986 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8c6dp\"" Sep 11 05:21:30.492005 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.491981 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c"] Sep 11 05:21:30.507804 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.507772 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m79z4"] Sep 11 05:21:30.509783 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.509765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.511996 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.511969 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 05:21:30.512145 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.512062 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 05:21:30.512145 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.512073 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 05:21:30.512259 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.512154 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4z67m\"" Sep 11 05:21:30.525491 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d9d622e-ff22-47a3-93e7-a75acf4c2500-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.525491 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.525689 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.525689 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-accelerators-collector-config\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.525689 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-wtmp\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.525689 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67cj4\" (UniqueName: \"kubernetes.io/projected/c896934d-7953-4886-b378-0db8bf0f757a-kube-api-access-67cj4\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.525814 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d57b147a-1998-4dd1-af68-f081b962d8af-metrics-client-ca\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.525814 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-textfile\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.525814 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw9l5\" (UniqueName: \"kubernetes.io/projected/d57b147a-1998-4dd1-af68-f081b962d8af-kube-api-access-jw9l5\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.525814 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d9d622e-ff22-47a3-93e7-a75acf4c2500-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.525938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.525938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525904 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d57b147a-1998-4dd1-af68-f081b962d8af-root\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.525938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525921 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57b147a-1998-4dd1-af68-f081b962d8af-sys\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.526040 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.526040 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.525975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brd6\" (UniqueName: \"kubernetes.io/projected/7d9d622e-ff22-47a3-93e7-a75acf4c2500-kube-api-access-8brd6\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.526040 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.526000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c896934d-7953-4886-b378-0db8bf0f757a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.526135 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.526055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-tls\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.526135 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.526089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c896934d-7953-4886-b378-0db8bf0f757a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.526135 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.526124 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d9d622e-ff22-47a3-93e7-a75acf4c2500-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.627325 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d57b147a-1998-4dd1-af68-f081b962d8af-metrics-client-ca\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.627325 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-textfile\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw9l5\" (UniqueName: \"kubernetes.io/projected/d57b147a-1998-4dd1-af68-f081b962d8af-kube-api-access-jw9l5\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d9d622e-ff22-47a3-93e7-a75acf4c2500-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d57b147a-1998-4dd1-af68-f081b962d8af-root\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57b147a-1998-4dd1-af68-f081b962d8af-sys\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627463 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brd6\" (UniqueName: \"kubernetes.io/projected/7d9d622e-ff22-47a3-93e7-a75acf4c2500-kube-api-access-8brd6\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c896934d-7953-4886-b378-0db8bf0f757a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627528 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d57b147a-1998-4dd1-af68-f081b962d8af-root\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-tls\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.627625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c896934d-7953-4886-b378-0db8bf0f757a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.628167 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d9d622e-ff22-47a3-93e7-a75acf4c2500-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.628167 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d9d622e-ff22-47a3-93e7-a75acf4c2500-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.628167 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:30.627713 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 05:21:30.628167 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.628167 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:30.627775 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-tls podName:d57b147a-1998-4dd1-af68-f081b962d8af nodeName:}" failed. No retries permitted until 2026-09-11 05:21:31.12775344 +0000 UTC m=+64.237491597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-tls") pod "node-exporter-m79z4" (UID: "d57b147a-1998-4dd1-af68-f081b962d8af") : secret "node-exporter-tls" not found Sep 11 05:21:30.628167 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:30.627807 2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 11 05:21:30.628167 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:30.627855 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-tls podName:c896934d-7953-4886-b378-0db8bf0f757a nodeName:}" failed. No retries permitted until 2026-09-11 05:21:31.127843021 +0000 UTC m=+64.237581170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-bgz4c" (UID: "c896934d-7953-4886-b378-0db8bf0f757a") : secret "kube-state-metrics-tls" not found Sep 11 05:21:30.628167 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.627803 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-textfile\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.628565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628205 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57b147a-1998-4dd1-af68-f081b962d8af-sys\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.628565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.628565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628303 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c896934d-7953-4886-b378-0db8bf0f757a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.628565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-accelerators-collector-config\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.628565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d57b147a-1998-4dd1-af68-f081b962d8af-metrics-client-ca\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.628565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-wtmp\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.628565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67cj4\" (UniqueName: \"kubernetes.io/projected/c896934d-7953-4886-b378-0db8bf0f757a-kube-api-access-67cj4\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.628565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628481 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d9d622e-ff22-47a3-93e7-a75acf4c2500-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.628948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628600 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c896934d-7953-4886-b378-0db8bf0f757a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.628948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628620 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-wtmp\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.628948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.628816 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-accelerators-collector-config\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.629208 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.629187 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.630299 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.630279 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d9d622e-ff22-47a3-93e7-a75acf4c2500-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.630689 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.630670 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.630785 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.630716 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.630785 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.630732 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d9d622e-ff22-47a3-93e7-a75acf4c2500-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.641649 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.641624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw9l5\" (UniqueName: \"kubernetes.io/projected/d57b147a-1998-4dd1-af68-f081b962d8af-kube-api-access-jw9l5\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:30.641775 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.641752 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brd6\" (UniqueName: \"kubernetes.io/projected/7d9d622e-ff22-47a3-93e7-a75acf4c2500-kube-api-access-8brd6\") pod \"openshift-state-metrics-67dcf8c979-9j4z8\" (UID: \"7d9d622e-ff22-47a3-93e7-a75acf4c2500\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:30.642747 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.642731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67cj4\" (UniqueName: \"kubernetes.io/projected/c896934d-7953-4886-b378-0db8bf0f757a-kube-api-access-67cj4\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:30.655289 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.655260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597467cd95-qfpks" event={"ID":"b2758633-5ec4-4b49-91a9-5db89f072899","Type":"ContainerStarted","Data":"c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7"} Sep 11 05:21:30.681405 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.681347 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-597467cd95-qfpks" podStartSLOduration=1.690409187 podStartE2EDuration="4.681330252s" totalImagesPullingTime="2.990921065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:30.680341328 +0000 UTC m=+63.790079502" watchObservedRunningTime="2026-09-11 05:21:30.681330252 +0000 UTC m=+63.791068417" Sep 11 05:21:30.760873 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:30.760839 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" Sep 11 05:21:31.029067 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.028977 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8"] Sep 11 05:21:31.031693 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:31.031655 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9d622e_ff22_47a3_93e7_a75acf4c2500.slice/crio-fc61523831d9b9383826839b334d8341df23bda073ce16b7892ca1e86bd0cc7d WatchSource:0}: Error finding container fc61523831d9b9383826839b334d8341df23bda073ce16b7892ca1e86bd0cc7d: Status 404 returned error can't find the container with id fc61523831d9b9383826839b334d8341df23bda073ce16b7892ca1e86bd0cc7d Sep 11 05:21:31.132850 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.132825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-tls\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:31.132964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.132878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:31.135200 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.135166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d57b147a-1998-4dd1-af68-f081b962d8af-node-exporter-tls\") pod \"node-exporter-m79z4\" (UID: \"d57b147a-1998-4dd1-af68-f081b962d8af\") " pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:31.135308 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.135281 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c896934d-7953-4886-b378-0db8bf0f757a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bgz4c\" (UID: \"c896934d-7953-4886-b378-0db8bf0f757a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:31.343862 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.343780 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-754d7dd4f7-2bfgv"] Sep 11 05:21:31.345680 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.345662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.354852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.354825 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 05:21:31.363531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.363503 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-754d7dd4f7-2bfgv"] Sep 11 05:21:31.393288 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.393246 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" Sep 11 05:21:31.418105 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.418074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m79z4" Sep 11 05:21:31.428691 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:31.428659 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57b147a_1998_4dd1_af68_f081b962d8af.slice/crio-559d561e182fb839f03a563cafe78b3fff2db9e5215eb79021bc35c5ba2cf9ac WatchSource:0}: Error finding container 559d561e182fb839f03a563cafe78b3fff2db9e5215eb79021bc35c5ba2cf9ac: Status 404 returned error can't find the container with id 559d561e182fb839f03a563cafe78b3fff2db9e5215eb79021bc35c5ba2cf9ac Sep 11 05:21:31.435724 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.435688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-oauth-config\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.435882 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.435750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-console-config\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.435882 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.435827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnx95\" (UniqueName: \"kubernetes.io/projected/acd60a94-c98a-409f-b438-f638dda6c177-kube-api-access-tnx95\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.435997 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.435909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-oauth-serving-cert\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.435997 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.435945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-service-ca\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.436095 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.436015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-serving-cert\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.436095 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.436041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-trusted-ca-bundle\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.537223 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.537184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-serving-cert\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.537223 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.537224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-trusted-ca-bundle\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.537818 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.537267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-oauth-config\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.537818 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.537291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-console-config\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.537818 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.537309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnx95\" (UniqueName: \"kubernetes.io/projected/acd60a94-c98a-409f-b438-f638dda6c177-kube-api-access-tnx95\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.537818 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.537333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-oauth-serving-cert\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.537818 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.537358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-service-ca\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.538251 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.538198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-console-config\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.538371 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.538263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-oauth-serving-cert\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.538371 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.538345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-trusted-ca-bundle\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.538496 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.538412 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-service-ca\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.539919 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.539895 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-oauth-config\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.540091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.540057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-serving-cert\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.544918 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.544899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnx95\" (UniqueName: \"kubernetes.io/projected/acd60a94-c98a-409f-b438-f638dda6c177-kube-api-access-tnx95\") pod \"console-754d7dd4f7-2bfgv\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.555956 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.555927 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 05:21:31.559049 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.559025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.563015 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.562771 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 05:21:31.563015 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.562818 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 05:21:31.563015 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.562846 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 05:21:31.563015 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.562867 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 05:21:31.563015 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.562880 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 05:21:31.563015 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.562890 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-llxmw\"" Sep 11 05:21:31.563447 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.563144 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 05:21:31.563447 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.563214 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 05:21:31.563447 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.563224 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 05:21:31.563447 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.563218 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 05:21:31.579988 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.579960 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 05:21:31.638062 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.637973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-tls-assets\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638062 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638062 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-volume\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638287 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638287 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638287 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638287 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638287 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-out\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638287 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638287 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638504 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-web-config\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638504 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx52f\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-kube-api-access-vx52f\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.638504 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.638429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.655009 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.654975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:31.661260 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.661221 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" event={"ID":"7d9d622e-ff22-47a3-93e7-a75acf4c2500","Type":"ContainerStarted","Data":"b533fd6aeb38c658cfef80df5295a717e998bc8d44b0e2c32c813cd078735c2c"} Sep 11 05:21:31.661400 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.661302 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" event={"ID":"7d9d622e-ff22-47a3-93e7-a75acf4c2500","Type":"ContainerStarted","Data":"c8fe2e2d0e50b40cd59e06663f917a258b89e2718eff9f3a8cb5a9bb167f1647"} Sep 11 05:21:31.661400 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.661332 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" event={"ID":"7d9d622e-ff22-47a3-93e7-a75acf4c2500","Type":"ContainerStarted","Data":"fc61523831d9b9383826839b334d8341df23bda073ce16b7892ca1e86bd0cc7d"} Sep 11 05:21:31.664222 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.664196 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m79z4" event={"ID":"d57b147a-1998-4dd1-af68-f081b962d8af","Type":"ContainerStarted","Data":"559d561e182fb839f03a563cafe78b3fff2db9e5215eb79021bc35c5ba2cf9ac"} Sep 11 05:21:31.668213 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.668187 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c"] Sep 11 05:21:31.671373 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:31.671344 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc896934d_7953_4886_b378_0db8bf0f757a.slice/crio-b127ec03d93c159dd9b87f1f72ea0585f29448bb90084f76db1721021aa52de3 WatchSource:0}: Error finding container b127ec03d93c159dd9b87f1f72ea0585f29448bb90084f76db1721021aa52de3: Status 404 returned error can't find the container with id b127ec03d93c159dd9b87f1f72ea0585f29448bb90084f76db1721021aa52de3 Sep 11 05:21:31.739459 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739663 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739663 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739663 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-out\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739663 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739663 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739663 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-web-config\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx52f\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-kube-api-access-vx52f\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-tls-assets\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.739952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-volume\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.740201 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.739976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.741030 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:21:31.740692 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-trusted-ca-bundle podName:591d4a15-b6f1-4ae5-ad8d-7a030da42272 nodeName:}" failed. No retries permitted until 2026-09-11 05:21:32.240669972 +0000 UTC m=+65.350408132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272") : configmap references non-existent config key: ca-bundle.crt Sep 11 05:21:31.741030 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.740737 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.741030 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.740994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.742979 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.742947 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.743370 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.743350 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-web-config\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.743461 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.743426 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-out\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.744050 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.743882 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.744050 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.743961 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.744178 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.744132 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-volume\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.744447 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.744424 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.744851 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.744829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.745303 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.745283 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-tls-assets\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.749159 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.749137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx52f\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-kube-api-access-vx52f\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:31.948707 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:31.929598 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-754d7dd4f7-2bfgv"] Sep 11 05:21:31.954787 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:31.954750 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd60a94_c98a_409f_b438_f638dda6c177.slice/crio-db2be6209cceef17e797398728d6460ea2b289a8cd739231b4e4010d5aa3185d WatchSource:0}: Error finding container db2be6209cceef17e797398728d6460ea2b289a8cd739231b4e4010d5aa3185d: Status 404 returned error can't find the container with id db2be6209cceef17e797398728d6460ea2b289a8cd739231b4e4010d5aa3185d Sep 11 05:21:32.244966 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.244884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:32.245638 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.245612 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:32.470085 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.470052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:21:32.555421 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.555324 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9"] Sep 11 05:21:32.565338 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.565308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.568769 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.567950 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5p4gz\"" Sep 11 05:21:32.568769 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.568378 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 11 05:21:32.569380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.569046 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 11 05:21:32.569380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.569197 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-asffapjirr7cs\"" Sep 11 05:21:32.569380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.569246 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 11 05:21:32.569380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.569268 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 11 05:21:32.569380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.569277 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 11 05:21:32.572924 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.572898 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9"] Sep 11 05:21:32.650614 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.650583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.650729 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.650655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqhmw\" (UniqueName: \"kubernetes.io/projected/48572d15-cecc-4119-8ae4-519cbd0b0b0a-kube-api-access-sqhmw\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.650803 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.650730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.650803 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.650791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.650898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.650879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.650951 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.650915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-tls\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.651007 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.650962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48572d15-cecc-4119-8ae4-519cbd0b0b0a-metrics-client-ca\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.651061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.651012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-grpc-tls\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.669576 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.669528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" event={"ID":"7d9d622e-ff22-47a3-93e7-a75acf4c2500","Type":"ContainerStarted","Data":"f9d37f179353bba761d96ae133ff1df9e8e80fe15402720dfab5186651def231"} Sep 11 05:21:32.670795 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.670768 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" event={"ID":"c896934d-7953-4886-b378-0db8bf0f757a","Type":"ContainerStarted","Data":"b127ec03d93c159dd9b87f1f72ea0585f29448bb90084f76db1721021aa52de3"} Sep 11 05:21:32.672222 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.672199 2577 generic.go:356] "Generic (PLEG): container finished" podID="d57b147a-1998-4dd1-af68-f081b962d8af" containerID="5968839a2c7d960dae448392c884fa247b83065420a9caa1185848f6211093cc" exitCode=0 Sep 11 05:21:32.672325 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.672267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m79z4" event={"ID":"d57b147a-1998-4dd1-af68-f081b962d8af","Type":"ContainerDied","Data":"5968839a2c7d960dae448392c884fa247b83065420a9caa1185848f6211093cc"} Sep 11 05:21:32.673897 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.673877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754d7dd4f7-2bfgv" event={"ID":"acd60a94-c98a-409f-b438-f638dda6c177","Type":"ContainerStarted","Data":"4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7"} Sep 11 05:21:32.673993 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.673914 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754d7dd4f7-2bfgv" event={"ID":"acd60a94-c98a-409f-b438-f638dda6c177","Type":"ContainerStarted","Data":"db2be6209cceef17e797398728d6460ea2b289a8cd739231b4e4010d5aa3185d"} Sep 11 05:21:32.689700 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.689645 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9j4z8" podStartSLOduration=1.327853759 podStartE2EDuration="2.689622063s" totalImagesPullingTime="1.361768304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:32.688210127 +0000 UTC m=+65.797948292" watchObservedRunningTime="2026-09-11 05:21:32.689622063 +0000 UTC m=+65.799360229" Sep 11 05:21:32.723779 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.723719 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-754d7dd4f7-2bfgv" podStartSLOduration=1.7237024459999999 podStartE2EDuration="1.723702446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:32.722571734 +0000 UTC m=+65.832309894" watchObservedRunningTime="2026-09-11 05:21:32.723702446 +0000 UTC m=+65.833440609" Sep 11 05:21:32.751791 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.751764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48572d15-cecc-4119-8ae4-519cbd0b0b0a-metrics-client-ca\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.752395 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.752034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-grpc-tls\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.752395 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.752141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.752395 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.752188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqhmw\" (UniqueName: \"kubernetes.io/projected/48572d15-cecc-4119-8ae4-519cbd0b0b0a-kube-api-access-sqhmw\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.752395 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.752226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.752395 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.752319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.753460 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.752766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.753460 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.753151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48572d15-cecc-4119-8ae4-519cbd0b0b0a-metrics-client-ca\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.753460 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.753204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-tls\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.755438 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.755026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-grpc-tls\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.755438 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.755383 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.755999 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.755953 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.755999 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.755959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-tls\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.756174 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.756155 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.756302 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.756282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48572d15-cecc-4119-8ae4-519cbd0b0b0a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.760102 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.760077 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqhmw\" (UniqueName: \"kubernetes.io/projected/48572d15-cecc-4119-8ae4-519cbd0b0b0a-kube-api-access-sqhmw\") pod \"thanos-querier-7b4d5ff898-l2gp9\" (UID: \"48572d15-cecc-4119-8ae4-519cbd0b0b0a\") " pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:32.780392 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.780364 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 05:21:32.786658 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:32.786626 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591d4a15_b6f1_4ae5_ad8d_7a030da42272.slice/crio-8e8c77cde682ceb088f84921324f30504c829a68007c71dd638ca984b8dd9878 WatchSource:0}: Error finding container 8e8c77cde682ceb088f84921324f30504c829a68007c71dd638ca984b8dd9878: Status 404 returned error can't find the container with id 8e8c77cde682ceb088f84921324f30504c829a68007c71dd638ca984b8dd9878 Sep 11 05:21:32.877938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:32.877901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:33.056516 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.056473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:33.056710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.056595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:33.059297 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.059274 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 05:21:33.059348 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.059274 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 05:21:33.069328 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.069298 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540b042-b24e-416e-b30a-84346d4dd47f-original-pull-secret\") pod \"global-pull-secret-syncer-5fkvd\" (UID: \"a540b042-b24e-416e-b30a-84346d4dd47f\") " pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:33.069328 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.069315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac746c00-7462-4686-9f96-7d5be8804348-metrics-certs\") pod \"network-metrics-daemon-hqmnw\" (UID: \"ac746c00-7462-4686-9f96-7d5be8804348\") " pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:33.156761 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.156720 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9"] Sep 11 05:21:33.159362 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:33.159329 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48572d15_cecc_4119_8ae4_519cbd0b0b0a.slice/crio-8497886083ed255f4dc72d521cdb96bfd667f7cce482e5bc50c2a85edd4f9056 WatchSource:0}: Error finding container 8497886083ed255f4dc72d521cdb96bfd667f7cce482e5bc50c2a85edd4f9056: Status 404 returned error can't find the container with id 8497886083ed255f4dc72d521cdb96bfd667f7cce482e5bc50c2a85edd4f9056 Sep 11 05:21:33.258206 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.258150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:33.260771 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.260746 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 05:21:33.270787 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.270760 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 05:21:33.282504 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.282468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrsf\" (UniqueName: \"kubernetes.io/projected/e699639b-b57e-4d9a-93b9-ec9b71a507fc-kube-api-access-2hrsf\") pod \"network-check-target-g8zdr\" (UID: \"e699639b-b57e-4d9a-93b9-ec9b71a507fc\") " pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:33.361963 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.361881 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g92vd\"" Sep 11 05:21:33.366110 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.366066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5fkvd" Sep 11 05:21:33.370259 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.370231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqmnw" Sep 11 05:21:33.372407 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.372383 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dpzsh\"" Sep 11 05:21:33.380720 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.380695 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:33.649749 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.649661 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5fkvd"] Sep 11 05:21:33.675289 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.675195 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hqmnw"] Sep 11 05:21:33.677614 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.677590 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8zdr"] Sep 11 05:21:33.687751 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.687715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m79z4" event={"ID":"d57b147a-1998-4dd1-af68-f081b962d8af","Type":"ContainerStarted","Data":"1ba5a955a56543f752be5c3b52a6089aa018f6b6cee10f127246c3a4eb66b11d"} Sep 11 05:21:33.687885 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.687816 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m79z4" event={"ID":"d57b147a-1998-4dd1-af68-f081b962d8af","Type":"ContainerStarted","Data":"b81f5825531a3231db6be9905b44f8f3fcd16fac960da01b59ef515d8b7a64ad"} Sep 11 05:21:33.690059 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.690024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerStarted","Data":"8e8c77cde682ceb088f84921324f30504c829a68007c71dd638ca984b8dd9878"} Sep 11 05:21:33.692226 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.692187 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" event={"ID":"48572d15-cecc-4119-8ae4-519cbd0b0b0a","Type":"ContainerStarted","Data":"8497886083ed255f4dc72d521cdb96bfd667f7cce482e5bc50c2a85edd4f9056"} Sep 11 05:21:33.704617 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:33.704560 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m79z4" podStartSLOduration=2.571886428 podStartE2EDuration="3.704525706s" totalImagesPullingTime="1.068362664s" totalInitContainerRuntime="64.276614ms" isStatefulPod=true podCreationTimestamp="2026-09-11 05:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:33.704056186 +0000 UTC m=+66.813794355" watchObservedRunningTime="2026-09-11 05:21:33.704525706 +0000 UTC m=+66.814263876" Sep 11 05:21:33.914440 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:33.914360 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac746c00_7462_4686_9f96_7d5be8804348.slice/crio-d7fb0690bca860aceebb64610bea0f0040fe5f1e71b69b4db9cefc801d32d33e WatchSource:0}: Error finding container d7fb0690bca860aceebb64610bea0f0040fe5f1e71b69b4db9cefc801d32d33e: Status 404 returned error can't find the container with id d7fb0690bca860aceebb64610bea0f0040fe5f1e71b69b4db9cefc801d32d33e Sep 11 05:21:33.914813 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:33.914791 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda540b042_b24e_416e_b30a_84346d4dd47f.slice/crio-05e7dc9a68766eed52499bb240bede468c5f534ffdbd3ae0d27205a198b9ee55 WatchSource:0}: Error finding container 05e7dc9a68766eed52499bb240bede468c5f534ffdbd3ae0d27205a198b9ee55: Status 404 returned error can't find the container with id 05e7dc9a68766eed52499bb240bede468c5f534ffdbd3ae0d27205a198b9ee55 Sep 11 05:21:33.917261 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:33.917223 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode699639b_b57e_4d9a_93b9_ec9b71a507fc.slice/crio-a51efb1f31b3f3274976c6f406416d5f73c63e0bdeb343dab107cffee3424e1b WatchSource:0}: Error finding container a51efb1f31b3f3274976c6f406416d5f73c63e0bdeb343dab107cffee3424e1b: Status 404 returned error can't find the container with id a51efb1f31b3f3274976c6f406416d5f73c63e0bdeb343dab107cffee3424e1b Sep 11 05:21:34.631483 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.631455 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kw5tg" Sep 11 05:21:34.700713 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.700639 2577 generic.go:356] "Generic (PLEG): container finished" podID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerID="c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488" exitCode=0 Sep 11 05:21:34.701279 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.701173 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerDied","Data":"c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488"} Sep 11 05:21:34.703767 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.703735 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hqmnw" event={"ID":"ac746c00-7462-4686-9f96-7d5be8804348","Type":"ContainerStarted","Data":"d7fb0690bca860aceebb64610bea0f0040fe5f1e71b69b4db9cefc801d32d33e"} Sep 11 05:21:34.706413 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.706323 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" event={"ID":"c896934d-7953-4886-b378-0db8bf0f757a","Type":"ContainerStarted","Data":"eea4d51c8587e557f992bac9a14f69dcfaf92716eb8f5a57fa986a05b87043c3"} Sep 11 05:21:34.708899 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.708859 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5fkvd" event={"ID":"a540b042-b24e-416e-b30a-84346d4dd47f","Type":"ContainerStarted","Data":"05e7dc9a68766eed52499bb240bede468c5f534ffdbd3ae0d27205a198b9ee55"} Sep 11 05:21:34.711544 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.711500 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8zdr" event={"ID":"e699639b-b57e-4d9a-93b9-ec9b71a507fc","Type":"ContainerStarted","Data":"a51efb1f31b3f3274976c6f406416d5f73c63e0bdeb343dab107cffee3424e1b"} Sep 11 05:21:34.829141 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.827624 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d57747d66-gjmx4"] Sep 11 05:21:34.844594 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.843960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.848165 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.847737 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 05:21:34.848165 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.847878 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 05:21:34.848165 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.848039 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pmndw\"" Sep 11 05:21:34.848165 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.848111 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-214qqifod7f8q\"" Sep 11 05:21:34.848516 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.848199 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 05:21:34.848516 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.848467 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 05:21:34.850753 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.850688 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d57747d66-gjmx4"] Sep 11 05:21:34.875929 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.875879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b344eedc-9880-44ab-8228-5df3091d8924-secret-metrics-server-client-certs\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.876100 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.875939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b344eedc-9880-44ab-8228-5df3091d8924-audit-log\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.876100 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.875979 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdb5\" (UniqueName: \"kubernetes.io/projected/b344eedc-9880-44ab-8228-5df3091d8924-kube-api-access-2pdb5\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.876100 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.876013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b344eedc-9880-44ab-8228-5df3091d8924-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.876100 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.876084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b344eedc-9880-44ab-8228-5df3091d8924-metrics-server-audit-profiles\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.876322 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.876141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b344eedc-9880-44ab-8228-5df3091d8924-secret-metrics-server-tls\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.876322 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.876172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b344eedc-9880-44ab-8228-5df3091d8924-client-ca-bundle\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.977515 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.976721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b344eedc-9880-44ab-8228-5df3091d8924-metrics-server-audit-profiles\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.977515 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.976795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b344eedc-9880-44ab-8228-5df3091d8924-secret-metrics-server-tls\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.977515 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.976841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b344eedc-9880-44ab-8228-5df3091d8924-client-ca-bundle\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.977515 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.976874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b344eedc-9880-44ab-8228-5df3091d8924-secret-metrics-server-client-certs\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.977515 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.976904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b344eedc-9880-44ab-8228-5df3091d8924-audit-log\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.977515 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.976934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb5\" (UniqueName: \"kubernetes.io/projected/b344eedc-9880-44ab-8228-5df3091d8924-kube-api-access-2pdb5\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.977515 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.976960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b344eedc-9880-44ab-8228-5df3091d8924-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.977994 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.977829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b344eedc-9880-44ab-8228-5df3091d8924-metrics-server-audit-profiles\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.978300 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.978194 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b344eedc-9880-44ab-8228-5df3091d8924-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.978611 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.978497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b344eedc-9880-44ab-8228-5df3091d8924-audit-log\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.984266 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.981158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b344eedc-9880-44ab-8228-5df3091d8924-secret-metrics-server-client-certs\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.984266 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.982475 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b344eedc-9880-44ab-8228-5df3091d8924-client-ca-bundle\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.984266 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.984187 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b344eedc-9880-44ab-8228-5df3091d8924-secret-metrics-server-tls\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:34.986831 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:34.986811 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdb5\" (UniqueName: \"kubernetes.io/projected/b344eedc-9880-44ab-8228-5df3091d8924-kube-api-access-2pdb5\") pod \"metrics-server-d57747d66-gjmx4\" (UID: \"b344eedc-9880-44ab-8228-5df3091d8924\") " pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:35.159239 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.159153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:35.251860 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.251483 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr"] Sep 11 05:21:35.268856 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.268822 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr"] Sep 11 05:21:35.269042 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.268987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:35.271786 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.271759 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 05:21:35.271995 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.271973 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hxmsb\"" Sep 11 05:21:35.383998 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.383960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-w7flr\" (UID: \"38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:35.384194 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.384025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rspzs\" (UniqueName: \"kubernetes.io/projected/38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad-kube-api-access-rspzs\") pod \"monitoring-plugin-5d85778b9d-w7flr\" (UID: \"38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:35.484972 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.484937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rspzs\" (UniqueName: \"kubernetes.io/projected/38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad-kube-api-access-rspzs\") pod \"monitoring-plugin-5d85778b9d-w7flr\" (UID: \"38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:35.485202 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.485108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-w7flr\" (UID: \"38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:35.487770 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.487740 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-w7flr\" (UID: \"38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:35.493854 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.493826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rspzs\" (UniqueName: \"kubernetes.io/projected/38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad-kube-api-access-rspzs\") pod \"monitoring-plugin-5d85778b9d-w7flr\" (UID: \"38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:35.586440 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:35.586399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:36.359442 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.359262 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d57747d66-gjmx4"] Sep 11 05:21:36.361005 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.360895 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr"] Sep 11 05:21:36.364490 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:36.364443 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb344eedc_9880_44ab_8228_5df3091d8924.slice/crio-2dd62c6185fbe3ae2bf45ac113fa9296853a6eb89cfa18bcb77dd0e208e2aa3c WatchSource:0}: Error finding container 2dd62c6185fbe3ae2bf45ac113fa9296853a6eb89cfa18bcb77dd0e208e2aa3c: Status 404 returned error can't find the container with id 2dd62c6185fbe3ae2bf45ac113fa9296853a6eb89cfa18bcb77dd0e208e2aa3c Sep 11 05:21:36.631102 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.631059 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:36.631289 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.631125 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:36.638166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.638136 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:36.725443 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.725347 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" event={"ID":"b344eedc-9880-44ab-8228-5df3091d8924","Type":"ContainerStarted","Data":"2dd62c6185fbe3ae2bf45ac113fa9296853a6eb89cfa18bcb77dd0e208e2aa3c"} Sep 11 05:21:36.731613 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.731574 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hqmnw" event={"ID":"ac746c00-7462-4686-9f96-7d5be8804348","Type":"ContainerStarted","Data":"fb0981e24d1fb13b5111678c19f741d7c236d92300d2e2d12aec5d6ea82b9c3a"} Sep 11 05:21:36.731784 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.731646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hqmnw" event={"ID":"ac746c00-7462-4686-9f96-7d5be8804348","Type":"ContainerStarted","Data":"16e778f052a2d107a34dfab7a9218bfa6b83ceff33572f7f80a2a98c4d4ae8e3"} Sep 11 05:21:36.735625 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.735539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" event={"ID":"38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad","Type":"ContainerStarted","Data":"4a6a153cf8ac9a57e47746ae1091322775500eb773ddfc82c0322f7977ec33ae"} Sep 11 05:21:36.741223 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.740534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" event={"ID":"c896934d-7953-4886-b378-0db8bf0f757a","Type":"ContainerStarted","Data":"c4f77a68091bb859d5157de96be7984768fbc83c0796be70b6625365ca99f496"} Sep 11 05:21:36.741223 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.740666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" event={"ID":"c896934d-7953-4886-b378-0db8bf0f757a","Type":"ContainerStarted","Data":"9ab7a7d4de935077a21c7a7bd1664af9d8a321618689aa8137d23b8c85b6087d"} Sep 11 05:21:36.748082 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.746094 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" event={"ID":"48572d15-cecc-4119-8ae4-519cbd0b0b0a","Type":"ContainerStarted","Data":"33b226c6797898aba08b03c0854e06ea3271b39c88f386ad40cd65f734b6fe25"} Sep 11 05:21:36.748082 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.746184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" event={"ID":"48572d15-cecc-4119-8ae4-519cbd0b0b0a","Type":"ContainerStarted","Data":"7eece6f56dbbe50be39a2e8f3f01e3f3ef498d7409471f9aeff3ecfdb6672286"} Sep 11 05:21:36.748082 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.746232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" event={"ID":"48572d15-cecc-4119-8ae4-519cbd0b0b0a","Type":"ContainerStarted","Data":"8de0c5b8326c828ac08a6a7af81efec2b5a42fb84fab63fae3ab2cf366c59ff6"} Sep 11 05:21:36.755002 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.752626 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hqmnw" podStartSLOduration=66.592819126 podStartE2EDuration="1m8.752604791s" totalImagesPullingTime="2.159785665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:36.750447887 +0000 UTC m=+69.860186055" watchObservedRunningTime="2026-09-11 05:21:36.752604791 +0000 UTC m=+69.862342961" Sep 11 05:21:36.755002 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.753165 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 05:21:36.756702 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.756662 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:21:36.756844 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.756771 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.760756 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.759653 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 05:21:36.760756 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.759868 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 05:21:36.760756 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.760130 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 05:21:36.760756 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.760312 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 05:21:36.760756 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.760494 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 05:21:36.764575 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.763651 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 05:21:36.764575 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.763771 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1tu3hf2k1u0e4\"" Sep 11 05:21:36.764575 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.764049 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 05:21:36.764575 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.764358 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 05:21:36.767859 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.765109 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 05:21:36.767859 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.765518 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 05:21:36.767859 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.766122 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 05:21:36.767859 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.766890 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 05:21:36.767859 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.767224 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2kn98\"" Sep 11 05:21:36.775195 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.768391 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 05:21:36.778492 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.777524 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 05:21:36.778492 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.777535 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 05:21:36.778492 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.778117 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bgz4c" podStartSLOduration=3.870911855 podStartE2EDuration="6.778096852s" totalImagesPullingTime="2.907184997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:36.77313908 +0000 UTC m=+69.882877247" watchObservedRunningTime="2026-09-11 05:21:36.778096852 +0000 UTC m=+69.887835026" Sep 11 05:21:36.798211 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.798176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.798801 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.798770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58rg\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-kube-api-access-x58rg\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.798801 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.798833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799042 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.798863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-config\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799042 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.798889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799042 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.798914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799042 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.798941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799042 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.798978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799042 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799331 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799331 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799331 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799331 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799331 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799331 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799665 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799374 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799665 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799461 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799665 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799665 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.799665 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.799544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900122 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900578 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900578 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900578 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900578 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900578 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900578 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900578 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900578 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.900578 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58rg\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-kube-api-access-x58rg\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.901078 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.901078 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-config\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.901078 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.901078 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.901078 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.900696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.905669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.901344 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.905669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.902116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.905669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.903146 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.905669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.904093 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.905669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.904757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.905669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.905041 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.905669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.905234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.906573 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.906525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.908888 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.907275 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.910919 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.910844 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.911584 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.911525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58rg\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-kube-api-access-x58rg\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.913750 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.913700 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.914372 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.914325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.914515 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.914489 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.914757 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.914733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.915343 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.915291 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.915956 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.915909 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.916707 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.916664 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.917351 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.917305 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-config\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:36.919754 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:36.919712 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:37.093298 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:37.093184 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:40.514076 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:40.514033 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-754d7dd4f7-2bfgv"] Sep 11 05:21:41.329626 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.329257 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 05:21:41.332126 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:21:41.331805 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8790e2_5bd0_40bc_93d7_26787795a981.slice/crio-d9ff6fda40eaaa033cd373526511daad25394d259efd54325528ccf88b425082 WatchSource:0}: Error finding container d9ff6fda40eaaa033cd373526511daad25394d259efd54325528ccf88b425082: Status 404 returned error can't find the container with id d9ff6fda40eaaa033cd373526511daad25394d259efd54325528ccf88b425082 Sep 11 05:21:41.655067 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.655039 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:21:41.779892 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.779857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8zdr" event={"ID":"e699639b-b57e-4d9a-93b9-ec9b71a507fc","Type":"ContainerStarted","Data":"572e7ec3835982857e3151d275299592b7359296d15f23a3028f11348eebbf44"} Sep 11 05:21:41.780095 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.779938 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:21:41.781610 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.781581 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" event={"ID":"b344eedc-9880-44ab-8228-5df3091d8924","Type":"ContainerStarted","Data":"2b9400239d231d00fd8e0930d01b773f58ac7c38c30671802fac41b6a51d3352"} Sep 11 05:21:41.785345 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.785317 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerStarted","Data":"4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb"} Sep 11 05:21:41.785485 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.785386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerStarted","Data":"62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5"} Sep 11 05:21:41.785485 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.785430 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerStarted","Data":"e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1"} Sep 11 05:21:41.785485 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.785468 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerStarted","Data":"b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc"} Sep 11 05:21:41.785666 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.785511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerStarted","Data":"6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55"} Sep 11 05:21:41.785666 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.785573 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerStarted","Data":"d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b"} Sep 11 05:21:41.786937 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.786913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" event={"ID":"38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad","Type":"ContainerStarted","Data":"f281a9ff69334c649e59be96fa1caf3ee941ca3c2cbd3f093397e337652d87d6"} Sep 11 05:21:41.787080 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.787065 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:41.790608 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.790578 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" event={"ID":"48572d15-cecc-4119-8ae4-519cbd0b0b0a","Type":"ContainerStarted","Data":"d312d56a0a9462bdf1ab56b25ff025f278af856ad8d6b96f1d1c884c5913a5e2"} Sep 11 05:21:41.790730 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.790641 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" event={"ID":"48572d15-cecc-4119-8ae4-519cbd0b0b0a","Type":"ContainerStarted","Data":"863d9b827f48e47f4434eb3c07663432b86d298f751f9ab3fe8830ee13c05436"} Sep 11 05:21:41.790730 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.790682 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" event={"ID":"48572d15-cecc-4119-8ae4-519cbd0b0b0a","Type":"ContainerStarted","Data":"0a6d0d308bf747d3d501286f9062154dcb45b6c7c3d379212275ad186ea98ff4"} Sep 11 05:21:41.790730 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.790727 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:41.792920 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.792898 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" Sep 11 05:21:41.793032 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.792980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5fkvd" event={"ID":"a540b042-b24e-416e-b30a-84346d4dd47f","Type":"ContainerStarted","Data":"3176bca211e7a993d59c21acc41199819e73c49cb12be2e32e985f47d84cccda"} Sep 11 05:21:41.797369 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.796484 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g8zdr" podStartSLOduration=66.523395743 podStartE2EDuration="1m13.79646576s" totalImagesPullingTime="7.273070017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:41.794416402 +0000 UTC m=+74.904154569" watchObservedRunningTime="2026-09-11 05:21:41.79646576 +0000 UTC m=+74.906203929" Sep 11 05:21:41.798590 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.798460 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerID="229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1" exitCode=0 Sep 11 05:21:41.798590 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.798537 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerDied","Data":"229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1"} Sep 11 05:21:41.798756 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.798634 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerStarted","Data":"d9ff6fda40eaaa033cd373526511daad25394d259efd54325528ccf88b425082"} Sep 11 05:21:41.799020 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.799004 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" Sep 11 05:21:41.809798 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.809721 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5fkvd" podStartSLOduration=66.530050721 podStartE2EDuration="1m13.809706185s" totalImagesPullingTime="7.279655464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:41.808334373 +0000 UTC m=+74.918072541" watchObservedRunningTime="2026-09-11 05:21:41.809706185 +0000 UTC m=+74.919444351" Sep 11 05:21:41.822780 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.822721 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-w7flr" podStartSLOduration=2.001056503 podStartE2EDuration="6.8227032s" totalImagesPullingTime="4.821646697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:41.822085308 +0000 UTC m=+74.931823475" watchObservedRunningTime="2026-09-11 05:21:41.8227032 +0000 UTC m=+74.932441368" Sep 11 05:21:41.847102 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.847049 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.157304533 podStartE2EDuration="10.847030911s" totalImagesPullingTime="7.668643361s" totalInitContainerRuntime="21.083017ms" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:41.844986038 +0000 UTC m=+74.954724240" watchObservedRunningTime="2026-09-11 05:21:41.847030911 +0000 UTC m=+74.956769078" Sep 11 05:21:41.865323 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.865267 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" podStartSLOduration=3.040431039 podStartE2EDuration="7.865235782s" totalImagesPullingTime="4.824804743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:41.864993269 +0000 UTC m=+74.974731439" watchObservedRunningTime="2026-09-11 05:21:41.865235782 +0000 UTC m=+74.974973928" Sep 11 05:21:41.891291 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:41.891230 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b4d5ff898-l2gp9" podStartSLOduration=2.280732669 podStartE2EDuration="9.891209988s" totalImagesPullingTime="7.610477319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:41.88970375 +0000 UTC m=+74.999441917" watchObservedRunningTime="2026-09-11 05:21:41.891209988 +0000 UTC m=+75.000948153" Sep 11 05:21:45.824075 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:45.824034 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerStarted","Data":"1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4"} Sep 11 05:21:45.824586 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:45.824112 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerStarted","Data":"c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c"} Sep 11 05:21:45.824586 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:45.824153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerStarted","Data":"72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57"} Sep 11 05:21:45.824586 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:45.824192 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerStarted","Data":"16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0"} Sep 11 05:21:45.824586 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:45.824269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerStarted","Data":"324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5"} Sep 11 05:21:45.824586 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:45.824314 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerStarted","Data":"6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83"} Sep 11 05:21:45.854380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:45.854329 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.684898121 podStartE2EDuration="9.854311589s" totalImagesPullingTime="3.134766827s" totalInitContainerRuntime="34.646641ms" isStatefulPod=false podCreationTimestamp="2026-09-11 05:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:21:45.852226153 +0000 UTC m=+78.961964330" watchObservedRunningTime="2026-09-11 05:21:45.854311589 +0000 UTC m=+78.964049754" Sep 11 05:21:47.093612 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:47.093562 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:21:51.397086 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:51.396966 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597467cd95-qfpks"] Sep 11 05:21:55.160109 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:55.160071 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:21:55.160489 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:21:55.160170 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:22:05.535262 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.535192 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-754d7dd4f7-2bfgv" podUID="acd60a94-c98a-409f-b438-f638dda6c177" containerName="console" containerID="cri-o://4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7" gracePeriod=15 Sep 11 05:22:05.778924 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.778892 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:22:05.880283 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880175 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-oauth-serving-cert\") pod \"acd60a94-c98a-409f-b438-f638dda6c177\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " Sep 11 05:22:05.880283 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880230 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-trusted-ca-bundle\") pod \"acd60a94-c98a-409f-b438-f638dda6c177\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " Sep 11 05:22:05.880283 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880252 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-service-ca\") pod \"acd60a94-c98a-409f-b438-f638dda6c177\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " Sep 11 05:22:05.880283 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880289 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnx95\" (UniqueName: \"kubernetes.io/projected/acd60a94-c98a-409f-b438-f638dda6c177-kube-api-access-tnx95\") pod \"acd60a94-c98a-409f-b438-f638dda6c177\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " Sep 11 05:22:05.880669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880316 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-oauth-config\") pod \"acd60a94-c98a-409f-b438-f638dda6c177\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " Sep 11 05:22:05.880669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880335 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-serving-cert\") pod \"acd60a94-c98a-409f-b438-f638dda6c177\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " Sep 11 05:22:05.880669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880371 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-console-config\") pod \"acd60a94-c98a-409f-b438-f638dda6c177\" (UID: \"acd60a94-c98a-409f-b438-f638dda6c177\") " Sep 11 05:22:05.880862 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880824 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "acd60a94-c98a-409f-b438-f638dda6c177" (UID: "acd60a94-c98a-409f-b438-f638dda6c177"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:05.880952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880831 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-service-ca" (OuterVolumeSpecName: "service-ca") pod "acd60a94-c98a-409f-b438-f638dda6c177" (UID: "acd60a94-c98a-409f-b438-f638dda6c177"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:05.880952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880833 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "acd60a94-c98a-409f-b438-f638dda6c177" (UID: "acd60a94-c98a-409f-b438-f638dda6c177"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:05.880952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.880923 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-console-config" (OuterVolumeSpecName: "console-config") pod "acd60a94-c98a-409f-b438-f638dda6c177" (UID: "acd60a94-c98a-409f-b438-f638dda6c177"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:05.883001 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.882974 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd60a94-c98a-409f-b438-f638dda6c177-kube-api-access-tnx95" (OuterVolumeSpecName: "kube-api-access-tnx95") pod "acd60a94-c98a-409f-b438-f638dda6c177" (UID: "acd60a94-c98a-409f-b438-f638dda6c177"). InnerVolumeSpecName "kube-api-access-tnx95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:22:05.883118 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.882977 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "acd60a94-c98a-409f-b438-f638dda6c177" (UID: "acd60a94-c98a-409f-b438-f638dda6c177"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:05.883118 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.883047 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "acd60a94-c98a-409f-b438-f638dda6c177" (UID: "acd60a94-c98a-409f-b438-f638dda6c177"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:05.910314 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.910280 2577 generic.go:356] "Generic (PLEG): container finished" podID="acd60a94-c98a-409f-b438-f638dda6c177" containerID="4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7" exitCode=0 Sep 11 05:22:05.910486 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.910329 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754d7dd4f7-2bfgv" Sep 11 05:22:05.910486 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.910339 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754d7dd4f7-2bfgv" event={"ID":"acd60a94-c98a-409f-b438-f638dda6c177","Type":"ContainerDied","Data":"4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7"} Sep 11 05:22:05.910486 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.910385 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754d7dd4f7-2bfgv" event={"ID":"acd60a94-c98a-409f-b438-f638dda6c177","Type":"ContainerDied","Data":"db2be6209cceef17e797398728d6460ea2b289a8cd739231b4e4010d5aa3185d"} Sep 11 05:22:05.910486 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.910422 2577 scope.go:117] "RemoveContainer" containerID="4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7" Sep 11 05:22:05.919742 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.919720 2577 scope.go:117] "RemoveContainer" containerID="4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7" Sep 11 05:22:05.920065 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:05.920044 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7\": container with ID starting with 4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7 not found: ID does not exist" containerID="4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7" Sep 11 05:22:05.920124 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.920074 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7"} err="failed to get container status \"4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7\": rpc error: code = NotFound desc = could not find container \"4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7\": container with ID starting with 4055f31272e8e1f403c3ecc661f92b0078d5434544fdb43329a181b9398e26f7 not found: ID does not exist" Sep 11 05:22:05.931284 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.931252 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-754d7dd4f7-2bfgv"] Sep 11 05:22:05.934770 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.934743 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-754d7dd4f7-2bfgv"] Sep 11 05:22:05.981566 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.981514 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-oauth-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:05.981566 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.981545 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-trusted-ca-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:05.981726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.981581 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-service-ca\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:05.981726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.981589 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnx95\" (UniqueName: \"kubernetes.io/projected/acd60a94-c98a-409f-b438-f638dda6c177-kube-api-access-tnx95\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:05.981726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.981599 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-oauth-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:05.981726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.981610 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd60a94-c98a-409f-b438-f638dda6c177-console-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:05.981726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:05.981618 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd60a94-c98a-409f-b438-f638dda6c177-console-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:07.355079 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:07.355041 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acd60a94-c98a-409f-b438-f638dda6c177" path="/var/lib/kubelet/pods/acd60a94-c98a-409f-b438-f638dda6c177/volumes" Sep 11 05:22:12.808107 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:12.808072 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g8zdr" Sep 11 05:22:15.167674 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:15.167640 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:22:15.171676 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:15.171654 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d57747d66-gjmx4" Sep 11 05:22:16.416871 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.416819 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-597467cd95-qfpks" podUID="b2758633-5ec4-4b49-91a9-5db89f072899" containerName="console" containerID="cri-o://c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7" gracePeriod=15 Sep 11 05:22:16.688129 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.688048 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:22:16.776477 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.776441 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-console-config\") pod \"b2758633-5ec4-4b49-91a9-5db89f072899\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " Sep 11 05:22:16.776661 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.776489 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-serving-cert\") pod \"b2758633-5ec4-4b49-91a9-5db89f072899\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " Sep 11 05:22:16.776661 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.776524 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-oauth-serving-cert\") pod \"b2758633-5ec4-4b49-91a9-5db89f072899\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " Sep 11 05:22:16.776661 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.776575 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwlw5\" (UniqueName: \"kubernetes.io/projected/b2758633-5ec4-4b49-91a9-5db89f072899-kube-api-access-rwlw5\") pod \"b2758633-5ec4-4b49-91a9-5db89f072899\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " Sep 11 05:22:16.776661 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.776630 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-service-ca\") pod \"b2758633-5ec4-4b49-91a9-5db89f072899\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " Sep 11 05:22:16.776848 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.776690 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-oauth-config\") pod \"b2758633-5ec4-4b49-91a9-5db89f072899\" (UID: \"b2758633-5ec4-4b49-91a9-5db89f072899\") " Sep 11 05:22:16.776904 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.776882 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-console-config" (OuterVolumeSpecName: "console-config") pod "b2758633-5ec4-4b49-91a9-5db89f072899" (UID: "b2758633-5ec4-4b49-91a9-5db89f072899"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:16.776995 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.776964 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-console-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:16.776995 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.776975 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b2758633-5ec4-4b49-91a9-5db89f072899" (UID: "b2758633-5ec4-4b49-91a9-5db89f072899"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:16.777144 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.777018 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-service-ca" (OuterVolumeSpecName: "service-ca") pod "b2758633-5ec4-4b49-91a9-5db89f072899" (UID: "b2758633-5ec4-4b49-91a9-5db89f072899"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:16.778955 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.778918 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2758633-5ec4-4b49-91a9-5db89f072899-kube-api-access-rwlw5" (OuterVolumeSpecName: "kube-api-access-rwlw5") pod "b2758633-5ec4-4b49-91a9-5db89f072899" (UID: "b2758633-5ec4-4b49-91a9-5db89f072899"). InnerVolumeSpecName "kube-api-access-rwlw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:22:16.778955 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.778929 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b2758633-5ec4-4b49-91a9-5db89f072899" (UID: "b2758633-5ec4-4b49-91a9-5db89f072899"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:16.779081 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.778966 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b2758633-5ec4-4b49-91a9-5db89f072899" (UID: "b2758633-5ec4-4b49-91a9-5db89f072899"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:16.878091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.878052 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:16.878091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.878082 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-oauth-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:16.878091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.878090 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwlw5\" (UniqueName: \"kubernetes.io/projected/b2758633-5ec4-4b49-91a9-5db89f072899-kube-api-access-rwlw5\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:16.878091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.878099 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2758633-5ec4-4b49-91a9-5db89f072899-service-ca\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:16.878366 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.878108 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2758633-5ec4-4b49-91a9-5db89f072899-console-oauth-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:16.958190 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.958106 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2758633-5ec4-4b49-91a9-5db89f072899" containerID="c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7" exitCode=0 Sep 11 05:22:16.958190 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.958162 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597467cd95-qfpks" Sep 11 05:22:16.958374 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.958165 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597467cd95-qfpks" event={"ID":"b2758633-5ec4-4b49-91a9-5db89f072899","Type":"ContainerDied","Data":"c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7"} Sep 11 05:22:16.958374 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.958271 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597467cd95-qfpks" event={"ID":"b2758633-5ec4-4b49-91a9-5db89f072899","Type":"ContainerDied","Data":"f4963f5c564a225aea808e0dc593f490917478cdecf6ffb728640ede6b6b4844"} Sep 11 05:22:16.958374 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.958292 2577 scope.go:117] "RemoveContainer" containerID="c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7" Sep 11 05:22:16.968045 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.968025 2577 scope.go:117] "RemoveContainer" containerID="c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7" Sep 11 05:22:16.968497 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:16.968451 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7\": container with ID starting with c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7 not found: ID does not exist" containerID="c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7" Sep 11 05:22:16.968497 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.968482 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7"} err="failed to get container status \"c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7\": rpc error: code = NotFound desc = could not find container \"c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7\": container with ID starting with c227d65a2ff5e3b6ca49387740839c10e77a284ad89680b28ffe80baeaf11cd7 not found: ID does not exist" Sep 11 05:22:16.978677 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.978650 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597467cd95-qfpks"] Sep 11 05:22:16.986601 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:16.986540 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-597467cd95-qfpks"] Sep 11 05:22:17.355275 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:17.355230 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2758633-5ec4-4b49-91a9-5db89f072899" path="/var/lib/kubelet/pods/b2758633-5ec4-4b49-91a9-5db89f072899/volumes" Sep 11 05:22:37.093529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:37.093492 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:37.113751 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:37.113725 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:38.060506 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:38.060476 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:50.733102 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:50.733070 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 05:22:50.733592 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:50.733368 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="alertmanager" containerID="cri-o://d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b" gracePeriod=120 Sep 11 05:22:50.733592 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:50.733422 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy-web" containerID="cri-o://b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc" gracePeriod=120 Sep 11 05:22:50.733592 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:50.733485 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy" containerID="cri-o://e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1" gracePeriod=120 Sep 11 05:22:50.733592 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:50.733476 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="config-reloader" containerID="cri-o://6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55" gracePeriod=120 Sep 11 05:22:50.733592 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:50.733434 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy-metric" containerID="cri-o://62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5" gracePeriod=120 Sep 11 05:22:50.733848 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:50.733485 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="prom-label-proxy" containerID="cri-o://4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb" gracePeriod=120 Sep 11 05:22:51.098728 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:51.098692 2577 generic.go:356] "Generic (PLEG): container finished" podID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerID="4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb" exitCode=0 Sep 11 05:22:51.098728 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:51.098718 2577 generic.go:356] "Generic (PLEG): container finished" podID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerID="e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1" exitCode=0 Sep 11 05:22:51.098728 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:51.098727 2577 generic.go:356] "Generic (PLEG): container finished" podID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerID="6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55" exitCode=0 Sep 11 05:22:51.098728 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:51.098733 2577 generic.go:356] "Generic (PLEG): container finished" podID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerID="d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b" exitCode=0 Sep 11 05:22:51.098990 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:51.098766 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerDied","Data":"4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb"} Sep 11 05:22:51.098990 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:51.098830 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerDied","Data":"e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1"} Sep 11 05:22:51.098990 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:51.098857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerDied","Data":"6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55"} Sep 11 05:22:51.098990 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:51.098884 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerDied","Data":"d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b"} Sep 11 05:22:51.986206 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:51.986183 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.079958 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.079884 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-metrics-client-ca\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.079958 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.079946 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-tls-assets\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.079965 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-volume\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.079998 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx52f\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-kube-api-access-vx52f\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080020 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-web\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080041 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-web-config\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080083 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-metric\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080162 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080112 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080449 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080175 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-main-db\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080449 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080211 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-out\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080449 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080245 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-main-tls\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080449 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080279 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-cluster-tls-config\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.080449 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080322 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-trusted-ca-bundle\") pod \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\" (UID: \"591d4a15-b6f1-4ae5-ad8d-7a030da42272\") " Sep 11 05:22:52.082518 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080270 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:52.082659 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080640 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:22:52.082659 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.080959 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:52.083360 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.083327 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-kube-api-access-vx52f" (OuterVolumeSpecName: "kube-api-access-vx52f") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "kube-api-access-vx52f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:22:52.083469 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.083391 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:22:52.083469 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.083405 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:52.083629 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.083567 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:52.083693 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.083663 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-volume" (OuterVolumeSpecName: "config-volume") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:52.083772 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.083747 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:52.084103 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.084086 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-out" (OuterVolumeSpecName: "config-out") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:22:52.085268 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.085248 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:52.088403 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.088360 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:52.093544 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.093524 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-web-config" (OuterVolumeSpecName: "web-config") pod "591d4a15-b6f1-4ae5-ad8d-7a030da42272" (UID: "591d4a15-b6f1-4ae5-ad8d-7a030da42272"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:52.106513 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.106489 2577 generic.go:356] "Generic (PLEG): container finished" podID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerID="62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5" exitCode=0 Sep 11 05:22:52.106513 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.106512 2577 generic.go:356] "Generic (PLEG): container finished" podID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerID="b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc" exitCode=0 Sep 11 05:22:52.106703 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.106595 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.106703 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.106590 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerDied","Data":"62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5"} Sep 11 05:22:52.106789 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.106703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerDied","Data":"b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc"} Sep 11 05:22:52.106789 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.106755 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"591d4a15-b6f1-4ae5-ad8d-7a030da42272","Type":"ContainerDied","Data":"8e8c77cde682ceb088f84921324f30504c829a68007c71dd638ca984b8dd9878"} Sep 11 05:22:52.106789 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.106775 2577 scope.go:117] "RemoveContainer" containerID="4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb" Sep 11 05:22:52.114602 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.114582 2577 scope.go:117] "RemoveContainer" containerID="62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5" Sep 11 05:22:52.122008 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.121989 2577 scope.go:117] "RemoveContainer" containerID="e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1" Sep 11 05:22:52.129045 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.129021 2577 scope.go:117] "RemoveContainer" containerID="b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc" Sep 11 05:22:52.130588 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.130545 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 05:22:52.136358 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.136342 2577 scope.go:117] "RemoveContainer" containerID="6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55" Sep 11 05:22:52.138844 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.138822 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 05:22:52.143807 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.143793 2577 scope.go:117] "RemoveContainer" containerID="d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b" Sep 11 05:22:52.150527 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.150511 2577 scope.go:117] "RemoveContainer" containerID="c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488" Sep 11 05:22:52.157516 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.157501 2577 scope.go:117] "RemoveContainer" containerID="4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb" Sep 11 05:22:52.157810 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:52.157791 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb\": container with ID starting with 4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb not found: ID does not exist" containerID="4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb" Sep 11 05:22:52.157869 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.157818 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb"} err="failed to get container status \"4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb\": rpc error: code = NotFound desc = could not find container \"4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb\": container with ID starting with 4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb not found: ID does not exist" Sep 11 05:22:52.157869 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.157837 2577 scope.go:117] "RemoveContainer" containerID="62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5" Sep 11 05:22:52.158099 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:52.158082 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5\": container with ID starting with 62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5 not found: ID does not exist" containerID="62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5" Sep 11 05:22:52.158159 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.158108 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5"} err="failed to get container status \"62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5\": rpc error: code = NotFound desc = could not find container \"62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5\": container with ID starting with 62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5 not found: ID does not exist" Sep 11 05:22:52.158159 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.158131 2577 scope.go:117] "RemoveContainer" containerID="e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1" Sep 11 05:22:52.158392 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:52.158374 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1\": container with ID starting with e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1 not found: ID does not exist" containerID="e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1" Sep 11 05:22:52.158432 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.158397 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1"} err="failed to get container status \"e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1\": rpc error: code = NotFound desc = could not find container \"e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1\": container with ID starting with e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1 not found: ID does not exist" Sep 11 05:22:52.158432 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.158414 2577 scope.go:117] "RemoveContainer" containerID="b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc" Sep 11 05:22:52.158686 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:52.158666 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc\": container with ID starting with b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc not found: ID does not exist" containerID="b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc" Sep 11 05:22:52.158762 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.158692 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc"} err="failed to get container status \"b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc\": rpc error: code = NotFound desc = could not find container \"b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc\": container with ID starting with b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc not found: ID does not exist" Sep 11 05:22:52.158762 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.158711 2577 scope.go:117] "RemoveContainer" containerID="6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55" Sep 11 05:22:52.158953 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:52.158938 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55\": container with ID starting with 6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55 not found: ID does not exist" containerID="6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55" Sep 11 05:22:52.158996 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.158957 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55"} err="failed to get container status \"6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55\": rpc error: code = NotFound desc = could not find container \"6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55\": container with ID starting with 6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55 not found: ID does not exist" Sep 11 05:22:52.158996 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.158969 2577 scope.go:117] "RemoveContainer" containerID="d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b" Sep 11 05:22:52.159228 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:52.159214 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b\": container with ID starting with d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b not found: ID does not exist" containerID="d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b" Sep 11 05:22:52.159277 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.159231 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b"} err="failed to get container status \"d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b\": rpc error: code = NotFound desc = could not find container \"d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b\": container with ID starting with d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b not found: ID does not exist" Sep 11 05:22:52.159277 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.159243 2577 scope.go:117] "RemoveContainer" containerID="c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488" Sep 11 05:22:52.159422 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:52.159407 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488\": container with ID starting with c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488 not found: ID does not exist" containerID="c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488" Sep 11 05:22:52.159457 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.159427 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488"} err="failed to get container status \"c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488\": rpc error: code = NotFound desc = could not find container \"c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488\": container with ID starting with c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488 not found: ID does not exist" Sep 11 05:22:52.159457 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.159438 2577 scope.go:117] "RemoveContainer" containerID="4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb" Sep 11 05:22:52.159679 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.159663 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb"} err="failed to get container status \"4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb\": rpc error: code = NotFound desc = could not find container \"4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb\": container with ID starting with 4d484087a3c909278f73ed5b9d5189231817c1c1cb1e2f962c1305ec9ee0b9cb not found: ID does not exist" Sep 11 05:22:52.159735 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.159680 2577 scope.go:117] "RemoveContainer" containerID="62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5" Sep 11 05:22:52.159876 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.159860 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5"} err="failed to get container status \"62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5\": rpc error: code = NotFound desc = could not find container \"62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5\": container with ID starting with 62706c55db6830cbd60f20eb5a20af089a006a45d4a6067fbcc8e30d53356df5 not found: ID does not exist" Sep 11 05:22:52.159917 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.159876 2577 scope.go:117] "RemoveContainer" containerID="e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1" Sep 11 05:22:52.160084 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.160066 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1"} err="failed to get container status \"e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1\": rpc error: code = NotFound desc = could not find container \"e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1\": container with ID starting with e0654a95961d6b3d6023570c732f9740639e7a6064070b648142e7a9c69eb9a1 not found: ID does not exist" Sep 11 05:22:52.160149 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.160086 2577 scope.go:117] "RemoveContainer" containerID="b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc" Sep 11 05:22:52.160300 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.160284 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc"} err="failed to get container status \"b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc\": rpc error: code = NotFound desc = could not find container \"b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc\": container with ID starting with b39d0090af006499aaf3433f2431cb926eb2b1b1f7f5721a59ec14ba8404bfbc not found: ID does not exist" Sep 11 05:22:52.160350 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.160300 2577 scope.go:117] "RemoveContainer" containerID="6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55" Sep 11 05:22:52.160498 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.160483 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55"} err="failed to get container status \"6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55\": rpc error: code = NotFound desc = could not find container \"6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55\": container with ID starting with 6cde6303a5d21c2d467bea2e38596fb7febdb1a2c4e3d8546c6cbcb87926bd55 not found: ID does not exist" Sep 11 05:22:52.160537 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.160497 2577 scope.go:117] "RemoveContainer" containerID="d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b" Sep 11 05:22:52.160721 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.160706 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b"} err="failed to get container status \"d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b\": rpc error: code = NotFound desc = could not find container \"d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b\": container with ID starting with d59b2dc8c4a9c301d82715a09ea735ad06f2736adcb65fa8be3b8c2230a8315b not found: ID does not exist" Sep 11 05:22:52.160763 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.160721 2577 scope.go:117] "RemoveContainer" containerID="c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488" Sep 11 05:22:52.160910 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.160894 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488"} err="failed to get container status \"c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488\": rpc error: code = NotFound desc = could not find container \"c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488\": container with ID starting with c2943be3ad1d12e830cc15801f21bf90ec02147d99e08539e96c0887ade88488 not found: ID does not exist" Sep 11 05:22:52.163585 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.163566 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 05:22:52.164056 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.163998 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="init-config-reloader" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164057 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="init-config-reloader" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164071 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy-web" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164077 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy-web" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164086 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2758633-5ec4-4b49-91a9-5db89f072899" containerName="console" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164093 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2758633-5ec4-4b49-91a9-5db89f072899" containerName="console" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164105 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="alertmanager" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164112 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="alertmanager" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164133 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="config-reloader" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164142 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="config-reloader" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164155 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy-metric" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164166 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy-metric" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164187 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="prom-label-proxy" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164196 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="prom-label-proxy" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164210 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acd60a94-c98a-409f-b438-f638dda6c177" containerName="console" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164220 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="acd60a94-c98a-409f-b438-f638dda6c177" containerName="console" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164233 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy" Sep 11 05:22:52.164236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164241 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy" Sep 11 05:22:52.164925 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164316 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2758633-5ec4-4b49-91a9-5db89f072899" containerName="console" Sep 11 05:22:52.164925 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164325 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy" Sep 11 05:22:52.164925 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164331 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="config-reloader" Sep 11 05:22:52.164925 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164339 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy-metric" Sep 11 05:22:52.164925 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164345 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="alertmanager" Sep 11 05:22:52.164925 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164353 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="prom-label-proxy" Sep 11 05:22:52.164925 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164360 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="acd60a94-c98a-409f-b438-f638dda6c177" containerName="console" Sep 11 05:22:52.164925 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.164369 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" containerName="kube-rbac-proxy-web" Sep 11 05:22:52.168245 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.168230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.170686 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.170665 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 05:22:52.170794 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.170667 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 05:22:52.170794 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.170733 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 05:22:52.170794 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.170787 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 05:22:52.170952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.170801 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 05:22:52.170952 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.170841 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-llxmw\"" Sep 11 05:22:52.171027 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.170952 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 05:22:52.171067 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.171025 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 05:22:52.171231 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.171218 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 05:22:52.175947 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.175928 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 05:22:52.181096 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5mr\" (UniqueName: \"kubernetes.io/projected/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-kube-api-access-js5mr\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181192 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-web-config\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181192 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181316 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181316 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181316 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181316 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-config-volume\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181517 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181517 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181517 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181507 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181692 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181586 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181692 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181692 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181683 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-config-out\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.181838 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181756 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx52f\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-kube-api-access-vx52f\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181838 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181774 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181838 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181789 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-web-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181939 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181838 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181939 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181861 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181939 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181877 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-main-db\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181939 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181889 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-out\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181939 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181901 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-secret-alertmanager-main-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181939 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181913 2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-cluster-tls-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181939 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181926 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.181939 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181939 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591d4a15-b6f1-4ae5-ad8d-7a030da42272-metrics-client-ca\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.182223 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181951 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/591d4a15-b6f1-4ae5-ad8d-7a030da42272-tls-assets\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.182223 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.181964 2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/591d4a15-b6f1-4ae5-ad8d-7a030da42272-config-volume\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:52.183338 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.183318 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 05:22:52.282539 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.282497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-config-out\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.282539 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.282541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5mr\" (UniqueName: \"kubernetes.io/projected/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-kube-api-access-js5mr\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.282812 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.282590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-web-config\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.282812 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.282613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.282812 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.282635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.282812 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.282650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.282812 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.282665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.282812 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.282682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-config-volume\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.283132 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.283104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.283196 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.283162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.283262 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.283203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.283262 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.283253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.283364 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.283316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.283692 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.283667 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.284321 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.284293 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.285145 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.285117 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.285701 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.285674 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.285701 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.285693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-config-out\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.285852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.285799 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-web-config\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.285852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.285807 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.286010 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.285869 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.286062 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.286017 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-config-volume\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.286326 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.286310 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.287140 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.287123 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.287487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.287472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.290327 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.290311 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5mr\" (UniqueName: \"kubernetes.io/projected/f71e5ddf-1f2c-49a1-be8d-7428d1cec010-kube-api-access-js5mr\") pod \"alertmanager-main-0\" (UID: \"f71e5ddf-1f2c-49a1-be8d-7428d1cec010\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.477722 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.477639 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 05:22:52.755763 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:52.755736 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 05:22:52.757063 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:22:52.757037 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71e5ddf_1f2c_49a1_be8d_7428d1cec010.slice/crio-6c5f774cc4940642ecaad5b3772420af72f5a32245151ab15cd854d2a82e6b42 WatchSource:0}: Error finding container 6c5f774cc4940642ecaad5b3772420af72f5a32245151ab15cd854d2a82e6b42: Status 404 returned error can't find the container with id 6c5f774cc4940642ecaad5b3772420af72f5a32245151ab15cd854d2a82e6b42 Sep 11 05:22:53.115072 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:53.114989 2577 generic.go:356] "Generic (PLEG): container finished" podID="f71e5ddf-1f2c-49a1-be8d-7428d1cec010" containerID="b9f07b3de835c518dd81f27fe345d85af547f0e3e7e669ab9ba22bf3f717750d" exitCode=0 Sep 11 05:22:53.115072 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:53.115042 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f71e5ddf-1f2c-49a1-be8d-7428d1cec010","Type":"ContainerDied","Data":"b9f07b3de835c518dd81f27fe345d85af547f0e3e7e669ab9ba22bf3f717750d"} Sep 11 05:22:53.115455 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:53.115092 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f71e5ddf-1f2c-49a1-be8d-7428d1cec010","Type":"ContainerStarted","Data":"6c5f774cc4940642ecaad5b3772420af72f5a32245151ab15cd854d2a82e6b42"} Sep 11 05:22:53.355637 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:53.355543 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="591d4a15-b6f1-4ae5-ad8d-7a030da42272" path="/var/lib/kubelet/pods/591d4a15-b6f1-4ae5-ad8d-7a030da42272/volumes" Sep 11 05:22:54.123689 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.123659 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f71e5ddf-1f2c-49a1-be8d-7428d1cec010","Type":"ContainerStarted","Data":"3239a24247e237841172556f03dbc89c49315031ca14f5e1c0e4826668b1f8c1"} Sep 11 05:22:54.124040 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.123718 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f71e5ddf-1f2c-49a1-be8d-7428d1cec010","Type":"ContainerStarted","Data":"e165b90fd6f5b48af440068bed88f5f92de7a6c3a81ccf6d934d2b993dffe0d2"} Sep 11 05:22:54.124040 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.123745 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f71e5ddf-1f2c-49a1-be8d-7428d1cec010","Type":"ContainerStarted","Data":"d087884a91b9906b2dec30f97c0b596c3431597c8b562bc20ac394e7848311f5"} Sep 11 05:22:54.124040 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.123769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f71e5ddf-1f2c-49a1-be8d-7428d1cec010","Type":"ContainerStarted","Data":"8f1cceee53b7eca9be2ffcce3c4f95ead4d51d97bb7cebcbc6529cab97b0282f"} Sep 11 05:22:54.124040 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.123793 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f71e5ddf-1f2c-49a1-be8d-7428d1cec010","Type":"ContainerStarted","Data":"dcbc945cc931e7482e2918862255a25bfdd43fc4f9f21858f3b5ee1351852524"} Sep 11 05:22:54.124040 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.123817 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f71e5ddf-1f2c-49a1-be8d-7428d1cec010","Type":"ContainerStarted","Data":"25ac89cc1e151f06840f69241d6cfd5b140ad565a2de46af54c0cd44a6afdcde"} Sep 11 05:22:54.151479 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.151414 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.139664556 podStartE2EDuration="2.151392515s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.727959ms" isStatefulPod=false podCreationTimestamp="2026-09-11 05:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:22:54.14866234 +0000 UTC m=+147.258400508" watchObservedRunningTime="2026-09-11 05:22:54.151392515 +0000 UTC m=+147.261130682" Sep 11 05:22:54.773615 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.773567 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-687649496d-nfrc9"] Sep 11 05:22:54.775979 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.775963 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.779255 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.778637 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 05:22:54.779255 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.778745 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ts2m6\"" Sep 11 05:22:54.779255 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.778913 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 05:22:54.779255 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.778957 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 05:22:54.779255 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.779084 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 05:22:54.779255 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.779134 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 05:22:54.783951 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.783875 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 05:22:54.792691 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.792666 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-687649496d-nfrc9"] Sep 11 05:22:54.806181 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.806152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6frc\" (UniqueName: \"kubernetes.io/projected/e11a7a4e-5c36-46e7-890b-be844875efb3-kube-api-access-p6frc\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.806317 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.806190 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e11a7a4e-5c36-46e7-890b-be844875efb3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.806317 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.806230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-telemeter-client-tls\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.806317 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.806250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.806317 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.806266 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-secret-telemeter-client\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.806317 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.806284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e11a7a4e-5c36-46e7-890b-be844875efb3-metrics-client-ca\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.806589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.806405 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e11a7a4e-5c36-46e7-890b-be844875efb3-serving-certs-ca-bundle\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.806589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.806447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-federate-client-tls\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.907585 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.907526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e11a7a4e-5c36-46e7-890b-be844875efb3-serving-certs-ca-bundle\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.907762 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.907596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-federate-client-tls\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.907762 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.907663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6frc\" (UniqueName: \"kubernetes.io/projected/e11a7a4e-5c36-46e7-890b-be844875efb3-kube-api-access-p6frc\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.907762 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.907706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e11a7a4e-5c36-46e7-890b-be844875efb3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.907762 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.907743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-telemeter-client-tls\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.909269 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.908532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.909269 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.908600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-secret-telemeter-client\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.909269 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.908639 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e11a7a4e-5c36-46e7-890b-be844875efb3-serving-certs-ca-bundle\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.909269 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.908636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e11a7a4e-5c36-46e7-890b-be844875efb3-metrics-client-ca\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.909269 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.908885 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e11a7a4e-5c36-46e7-890b-be844875efb3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.909269 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.909227 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e11a7a4e-5c36-46e7-890b-be844875efb3-metrics-client-ca\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.915586 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.911061 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-federate-client-tls\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.915586 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.911580 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.915586 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.912176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-secret-telemeter-client\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.916189 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.916162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e11a7a4e-5c36-46e7-890b-be844875efb3-telemeter-client-tls\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:54.918487 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:54.918457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6frc\" (UniqueName: \"kubernetes.io/projected/e11a7a4e-5c36-46e7-890b-be844875efb3-kube-api-access-p6frc\") pod \"telemeter-client-687649496d-nfrc9\" (UID: \"e11a7a4e-5c36-46e7-890b-be844875efb3\") " pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:55.063623 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:55.063518 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 05:22:55.063964 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:55.063933 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="prometheus" containerID="cri-o://6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83" gracePeriod=600 Sep 11 05:22:55.064048 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:55.063995 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4" gracePeriod=600 Sep 11 05:22:55.064107 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:55.064029 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="thanos-sidecar" containerID="cri-o://16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0" gracePeriod=600 Sep 11 05:22:55.064107 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:55.064044 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="config-reloader" containerID="cri-o://324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5" gracePeriod=600 Sep 11 05:22:55.064107 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:55.064083 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy" containerID="cri-o://c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c" gracePeriod=600 Sep 11 05:22:55.064308 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:55.064034 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy-web" containerID="cri-o://72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57" gracePeriod=600 Sep 11 05:22:55.086769 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:55.086746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" Sep 11 05:22:55.373426 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:55.373401 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-687649496d-nfrc9"] Sep 11 05:22:55.375470 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:22:55.375443 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11a7a4e_5c36_46e7_890b_be844875efb3.slice/crio-506b520a5bc737d41c24b9a47961f584433b97da7403753207629a0e97683290 WatchSource:0}: Error finding container 506b520a5bc737d41c24b9a47961f584433b97da7403753207629a0e97683290: Status 404 returned error can't find the container with id 506b520a5bc737d41c24b9a47961f584433b97da7403753207629a0e97683290 Sep 11 05:22:56.140669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140637 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerID="1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4" exitCode=0 Sep 11 05:22:56.140669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140664 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerID="c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c" exitCode=0 Sep 11 05:22:56.140669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140671 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerID="16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0" exitCode=0 Sep 11 05:22:56.140669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140676 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerID="324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5" exitCode=0 Sep 11 05:22:56.140961 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140682 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerID="6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83" exitCode=0 Sep 11 05:22:56.140961 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140714 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerDied","Data":"1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4"} Sep 11 05:22:56.140961 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140796 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerDied","Data":"c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c"} Sep 11 05:22:56.140961 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerDied","Data":"16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0"} Sep 11 05:22:56.140961 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140885 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerDied","Data":"324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5"} Sep 11 05:22:56.140961 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.140926 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerDied","Data":"6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83"} Sep 11 05:22:56.142460 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.142434 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" event={"ID":"e11a7a4e-5c36-46e7-890b-be844875efb3","Type":"ContainerStarted","Data":"506b520a5bc737d41c24b9a47961f584433b97da7403753207629a0e97683290"} Sep 11 05:22:56.330262 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.330233 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.425796 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-web-config\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.425847 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-trusted-ca-bundle\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.425883 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-config-out\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.425909 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-metrics-client-certs\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.425943 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-kubelet-serving-ca-bundle\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.425975 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-0\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-serving-certs-ca-bundle\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426050 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426081 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-db\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426108 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-metrics-client-ca\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426138 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-2\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426167 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-tls-assets\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426200 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-tls\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426240 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-1\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426289 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-grpc-tls\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426324 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x58rg\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-kube-api-access-x58rg\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426370 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.427091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426395 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-config\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.428402 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426418 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-kube-rbac-proxy\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.428402 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.426452 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-thanos-prometheus-http-client-file\") pod \"fb8790e2-5bd0-40bc-93d7-26787795a981\" (UID: \"fb8790e2-5bd0-40bc-93d7-26787795a981\") " Sep 11 05:22:56.428402 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.427382 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:56.428402 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.427961 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:56.428402 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.428112 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:56.428402 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.428168 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:56.428402 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.428246 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:56.430204 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.429176 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:56.430870 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.430796 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:22:56.432139 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.432099 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:22:56.433253 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.433210 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:56.433386 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.433363 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-kube-api-access-x58rg" (OuterVolumeSpecName: "kube-api-access-x58rg") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "kube-api-access-x58rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:22:56.433541 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.433519 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:56.433895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.433838 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:56.433976 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.433890 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:22:56.434045 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.433967 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-config-out" (OuterVolumeSpecName: "config-out") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:22:56.434045 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.433982 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:56.434297 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.434275 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-config" (OuterVolumeSpecName: "config") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:56.435236 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.435211 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:56.435608 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.435544 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:56.435812 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.435665 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:56.445384 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.445356 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-web-config" (OuterVolumeSpecName: "web-config") pod "fb8790e2-5bd0-40bc-93d7-26787795a981" (UID: "fb8790e2-5bd0-40bc-93d7-26787795a981"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:22:56.527346 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527310 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-config-out\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527346 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527340 2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-metrics-client-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527346 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527351 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527361 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527370 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527378 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527387 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-db\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527395 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-configmap-metrics-client-ca\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527403 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527412 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-tls-assets\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527424 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527437 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527451 2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-grpc-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527464 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x58rg\" (UniqueName: \"kubernetes.io/projected/fb8790e2-5bd0-40bc-93d7-26787795a981-kube-api-access-x58rg\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527472 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527482 2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527490 2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-secret-kube-rbac-proxy\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527498 2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527508 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb8790e2-5bd0-40bc-93d7-26787795a981-web-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:56.527651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:56.527519 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8790e2-5bd0-40bc-93d7-26787795a981-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:22:57.149608 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.149580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" event={"ID":"e11a7a4e-5c36-46e7-890b-be844875efb3","Type":"ContainerStarted","Data":"4ae3d7c28d443b274ca07fea9a90a03eb5e28c9954eadadca9b2e5c5f82068fe"} Sep 11 05:22:57.152833 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.152806 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerID="72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57" exitCode=0 Sep 11 05:22:57.152926 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.152862 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerDied","Data":"72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57"} Sep 11 05:22:57.152926 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.152876 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.152995 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.152935 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb8790e2-5bd0-40bc-93d7-26787795a981","Type":"ContainerDied","Data":"d9ff6fda40eaaa033cd373526511daad25394d259efd54325528ccf88b425082"} Sep 11 05:22:57.152995 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.152986 2577 scope.go:117] "RemoveContainer" containerID="1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4" Sep 11 05:22:57.163222 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.163198 2577 scope.go:117] "RemoveContainer" containerID="c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c" Sep 11 05:22:57.170584 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.170541 2577 scope.go:117] "RemoveContainer" containerID="72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57" Sep 11 05:22:57.177661 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.177635 2577 scope.go:117] "RemoveContainer" containerID="16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0" Sep 11 05:22:57.183827 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.183803 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 05:22:57.187869 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.187682 2577 scope.go:117] "RemoveContainer" containerID="324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5" Sep 11 05:22:57.188459 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.188435 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 05:22:57.199259 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.199241 2577 scope.go:117] "RemoveContainer" containerID="6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83" Sep 11 05:22:57.211391 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.211367 2577 scope.go:117] "RemoveContainer" containerID="229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1" Sep 11 05:22:57.213195 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213175 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 05:22:57.213632 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213614 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy-web" Sep 11 05:22:57.213722 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213634 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy-web" Sep 11 05:22:57.213722 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213649 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy-thanos" Sep 11 05:22:57.213722 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213657 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy-thanos" Sep 11 05:22:57.213722 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213670 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="init-config-reloader" Sep 11 05:22:57.213722 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213678 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="init-config-reloader" Sep 11 05:22:57.213722 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213691 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="prometheus" Sep 11 05:22:57.213722 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213699 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="prometheus" Sep 11 05:22:57.213722 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213718 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="thanos-sidecar" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213728 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="thanos-sidecar" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213741 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="config-reloader" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213749 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="config-reloader" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213761 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213769 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213845 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="config-reloader" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213856 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="thanos-sidecar" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213868 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy-thanos" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213880 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy-web" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213889 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="prometheus" Sep 11 05:22:57.214091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.213899 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" containerName="kube-rbac-proxy" Sep 11 05:22:57.217866 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.217848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.220161 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220142 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 05:22:57.220317 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220297 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 05:22:57.220434 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220419 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 05:22:57.220509 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220495 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 05:22:57.220594 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220518 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 05:22:57.220719 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220705 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 05:22:57.220786 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220771 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1tu3hf2k1u0e4\"" Sep 11 05:22:57.220832 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220774 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 05:22:57.220832 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220788 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 05:22:57.220832 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220796 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 05:22:57.220984 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.220970 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 05:22:57.221039 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.221031 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 05:22:57.221229 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.221213 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 05:22:57.221350 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.221337 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2kn98\"" Sep 11 05:22:57.223368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.223346 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 05:22:57.226209 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.226190 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 05:22:57.228524 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.228506 2577 scope.go:117] "RemoveContainer" containerID="1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4" Sep 11 05:22:57.228863 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:57.228833 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4\": container with ID starting with 1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4 not found: ID does not exist" containerID="1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4" Sep 11 05:22:57.228949 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.228868 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4"} err="failed to get container status \"1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4\": rpc error: code = NotFound desc = could not find container \"1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4\": container with ID starting with 1aaabcf1bd7258a32d0eb3b31c65a983401eb8f7e716420c843762caf7ae90b4 not found: ID does not exist" Sep 11 05:22:57.228949 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.228886 2577 scope.go:117] "RemoveContainer" containerID="c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c" Sep 11 05:22:57.229190 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:57.229172 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c\": container with ID starting with c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c not found: ID does not exist" containerID="c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c" Sep 11 05:22:57.229259 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.229199 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c"} err="failed to get container status \"c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c\": rpc error: code = NotFound desc = could not find container \"c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c\": container with ID starting with c711a7acb8dd1319f36804c7e84a1d880cb28ac4b9d70d3e6ccd140fd356a57c not found: ID does not exist" Sep 11 05:22:57.229259 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.229222 2577 scope.go:117] "RemoveContainer" containerID="72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57" Sep 11 05:22:57.229581 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:57.229539 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57\": container with ID starting with 72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57 not found: ID does not exist" containerID="72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57" Sep 11 05:22:57.229637 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.229583 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57"} err="failed to get container status \"72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57\": rpc error: code = NotFound desc = could not find container \"72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57\": container with ID starting with 72ca67748e5887c3001f0cb6f099b6ff8fad0056337fd10d5cbeea16547bcc57 not found: ID does not exist" Sep 11 05:22:57.229637 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.229606 2577 scope.go:117] "RemoveContainer" containerID="16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0" Sep 11 05:22:57.230620 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:57.230521 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0\": container with ID starting with 16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0 not found: ID does not exist" containerID="16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0" Sep 11 05:22:57.230620 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.230574 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0"} err="failed to get container status \"16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0\": rpc error: code = NotFound desc = could not find container \"16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0\": container with ID starting with 16c323091747f18ae332236596dce408526e595d0010d620ee0e6343e06659a0 not found: ID does not exist" Sep 11 05:22:57.230620 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.230595 2577 scope.go:117] "RemoveContainer" containerID="324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5" Sep 11 05:22:57.231083 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:57.230930 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5\": container with ID starting with 324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5 not found: ID does not exist" containerID="324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5" Sep 11 05:22:57.231178 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.231088 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5"} err="failed to get container status \"324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5\": rpc error: code = NotFound desc = could not find container \"324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5\": container with ID starting with 324edacfa00489f739a1461650140b77eda1f72c45ec9763820761c7729c24a5 not found: ID does not exist" Sep 11 05:22:57.231178 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.231110 2577 scope.go:117] "RemoveContainer" containerID="6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83" Sep 11 05:22:57.231948 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:57.231830 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83\": container with ID starting with 6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83 not found: ID does not exist" containerID="6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83" Sep 11 05:22:57.231948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.231859 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83"} err="failed to get container status \"6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83\": rpc error: code = NotFound desc = could not find container \"6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83\": container with ID starting with 6aa3cef2fcb7bce50c4657ff09591d2dd491920323213151a6ce863fe2bf1d83 not found: ID does not exist" Sep 11 05:22:57.231948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.231880 2577 scope.go:117] "RemoveContainer" containerID="229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1" Sep 11 05:22:57.232208 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:22:57.232186 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1\": container with ID starting with 229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1 not found: ID does not exist" containerID="229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1" Sep 11 05:22:57.232367 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.232215 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1"} err="failed to get container status \"229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1\": rpc error: code = NotFound desc = could not find container \"229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1\": container with ID starting with 229816f5f7d379ccbece406c06cfb688ce1d752454892fdb879b7178c8f2f0a1 not found: ID does not exist" Sep 11 05:22:57.233173 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.233154 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 05:22:57.333825 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.333795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-config-out\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334019 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.333837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334019 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.333860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334019 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.333877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-web-config\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334019 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.333946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334019 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.333978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334019 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.333997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334019 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lglp\" (UniqueName: \"kubernetes.io/projected/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-kube-api-access-5lglp\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334078 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334182 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-config\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.334768 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.334343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.358047 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.358016 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb8790e2-5bd0-40bc-93d7-26787795a981" path="/var/lib/kubelet/pods/fb8790e2-5bd0-40bc-93d7-26787795a981/volumes" Sep 11 05:22:57.435224 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435224 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435224 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lglp\" (UniqueName: \"kubernetes.io/projected/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-kube-api-access-5lglp\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435224 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435220 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-config\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-config-out\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-web-config\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.435853 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.436641 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.435887 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.436641 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.436008 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.436641 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.436121 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.436641 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.436145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.437912 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.437621 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.440047 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.440020 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.440684 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.440284 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-config-out\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.440684 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.440435 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.440684 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.440460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-config\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.440684 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.440467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.440684 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.440636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.440989 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.440782 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.440989 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.440913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-web-config\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.440989 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.440928 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.441160 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.441135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.441160 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.441154 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.441733 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.441704 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.442948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.442930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.443076 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.443055 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.443323 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.443308 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lglp\" (UniqueName: \"kubernetes.io/projected/7b9076e4-d0e2-4a09-a7d0-0012788e8eb4-kube-api-access-5lglp\") pod \"prometheus-k8s-0\" (UID: \"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.534470 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.534429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:22:57.813694 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:57.813656 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 05:22:57.815710 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:22:57.815690 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9076e4_d0e2_4a09_a7d0_0012788e8eb4.slice/crio-a3543194ac688d19aad15c95a904f1c18e0089328b94d3d0e36986938cbf078c WatchSource:0}: Error finding container a3543194ac688d19aad15c95a904f1c18e0089328b94d3d0e36986938cbf078c: Status 404 returned error can't find the container with id a3543194ac688d19aad15c95a904f1c18e0089328b94d3d0e36986938cbf078c Sep 11 05:22:58.160006 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:58.159913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" event={"ID":"e11a7a4e-5c36-46e7-890b-be844875efb3","Type":"ContainerStarted","Data":"a5c493f6b45b78d0c339746f5c1574ff8e5b5dc299a99dd37fafab64955d4a2e"} Sep 11 05:22:58.160006 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:58.159979 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" event={"ID":"e11a7a4e-5c36-46e7-890b-be844875efb3","Type":"ContainerStarted","Data":"d57c3ddb4ae9a60fd019b5dfdc1e6ee64174d0f4261f6285f54e6afccd1ad7d8"} Sep 11 05:22:58.161416 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:58.161393 2577 generic.go:356] "Generic (PLEG): container finished" podID="7b9076e4-d0e2-4a09-a7d0-0012788e8eb4" containerID="bd648c8c34a22ed78f7bf728132ba0a631a88c4a7240aad158724ff01a06768d" exitCode=0 Sep 11 05:22:58.161524 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:58.161430 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4","Type":"ContainerDied","Data":"bd648c8c34a22ed78f7bf728132ba0a631a88c4a7240aad158724ff01a06768d"} Sep 11 05:22:58.161524 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:58.161472 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4","Type":"ContainerStarted","Data":"a3543194ac688d19aad15c95a904f1c18e0089328b94d3d0e36986938cbf078c"} Sep 11 05:22:58.180343 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:58.180300 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-687649496d-nfrc9" podStartSLOduration=2.506816067 podStartE2EDuration="4.180282801s" totalImagesPullingTime="1.673466734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:22:58.179138433 +0000 UTC m=+151.288876602" watchObservedRunningTime="2026-09-11 05:22:58.180282801 +0000 UTC m=+151.290020961" Sep 11 05:22:59.171997 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:59.171953 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4","Type":"ContainerStarted","Data":"4ae2c64fc8d36b51c3b36552d84bc581b135b04d35d7674165398c14966997de"} Sep 11 05:22:59.172377 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:59.172060 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4","Type":"ContainerStarted","Data":"214677e56d9b4d6fe9f67b6ca72b0a51c36fd8bfccab72d290e683d03734bd0e"} Sep 11 05:22:59.172377 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:59.172089 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4","Type":"ContainerStarted","Data":"1e981e60aebf60557aaf0c8d0299c94f3c128a91395c93c96648e040ebe337fd"} Sep 11 05:22:59.172377 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:59.172124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4","Type":"ContainerStarted","Data":"4fc78e16a947e707f72cb9a484b4632c8450647e8bcc8aec5993dae53ed44877"} Sep 11 05:22:59.172377 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:59.172149 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4","Type":"ContainerStarted","Data":"8681f0df3fe7b0d1858d3d018eda9d86e31a325b888a1f8adde0076a0d323663"} Sep 11 05:22:59.172377 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:59.172173 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b9076e4-d0e2-4a09-a7d0-0012788e8eb4","Type":"ContainerStarted","Data":"f6b58b2d71f2df45787c848b536cd06a54fb4ae282eba71ba37687c1bd7deee0"} Sep 11 05:22:59.206046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:22:59.205990 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.182429728 podStartE2EDuration="2.205972057s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.542329ms" isStatefulPod=false podCreationTimestamp="2026-09-11 05:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:22:59.204469305 +0000 UTC m=+152.314207505" watchObservedRunningTime="2026-09-11 05:22:59.205972057 +0000 UTC m=+152.315710223" Sep 11 05:23:02.534707 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:23:02.534675 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:23:57.535710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:23:57.535672 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:23:57.551068 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:23:57.551040 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:23:58.426438 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:23:58.426405 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 05:25:27.272870 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:25:27.272848 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 05:26:42.879074 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.879029 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8"] Sep 11 05:26:42.882789 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.882772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:42.886183 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.886158 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 11 05:26:42.886298 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.886169 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 11 05:26:42.886372 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.886304 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 11 05:26:42.886453 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.886436 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-qnrd4\"" Sep 11 05:26:42.886531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.886518 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 05:26:42.887177 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.887157 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 11 05:26:42.887271 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.887196 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 11 05:26:42.887320 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.887301 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 05:26:42.897993 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:42.897973 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8"] Sep 11 05:26:43.023132 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.023093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-feast-data\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.023132 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.023133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gq22\" (UniqueName: \"kubernetes.io/projected/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-kube-api-access-8gq22\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.023370 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.023154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-online-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.023370 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.023216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-offline-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.023370 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.023247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-ui-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.023370 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.023290 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-registry-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.123948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.123911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-registry-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.124140 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.123979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-feast-data\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.124140 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.124005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gq22\" (UniqueName: \"kubernetes.io/projected/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-kube-api-access-8gq22\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.124140 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.124029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-online-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.124140 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.124076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-offline-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.124140 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.124111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-ui-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.124494 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.124462 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-feast-data\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.126524 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.126495 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-online-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.126647 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.126525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-registry-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.126691 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.126641 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-ui-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.126691 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.126655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-offline-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.135506 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.135459 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gq22\" (UniqueName: \"kubernetes.io/projected/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-kube-api-access-8gq22\") pod \"feast-simple-feast-setup-585fd8b5bc-vdxz8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.209163 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.209133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:26:43.486377 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.486349 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8"] Sep 11 05:26:43.487419 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:26:43.487396 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc47cdc32_592c_4cd1_9ab8_5d04e72a99e8.slice/crio-d33b3cf959d21331d1afead34c46433b213a80fa671d93badac70f08fee77ab0 WatchSource:0}: Error finding container d33b3cf959d21331d1afead34c46433b213a80fa671d93badac70f08fee77ab0: Status 404 returned error can't find the container with id d33b3cf959d21331d1afead34c46433b213a80fa671d93badac70f08fee77ab0 Sep 11 05:26:43.489281 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:43.489264 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 05:26:44.066277 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:26:44.066243 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerStarted","Data":"d33b3cf959d21331d1afead34c46433b213a80fa671d93badac70f08fee77ab0"} Sep 11 05:27:09.186049 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:09.185949 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerStarted","Data":"6b4507e084e1e7cc97139abef5ffcb148b6f10b028937ff0551a999015a37e50"} Sep 11 05:27:14.208728 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:14.208644 2577 generic.go:356] "Generic (PLEG): container finished" podID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerID="6b4507e084e1e7cc97139abef5ffcb148b6f10b028937ff0551a999015a37e50" exitCode=0 Sep 11 05:27:14.209179 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:14.208717 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerDied","Data":"6b4507e084e1e7cc97139abef5ffcb148b6f10b028937ff0551a999015a37e50"} Sep 11 05:27:15.214728 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:15.214692 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerStarted","Data":"f0c2191aadca2ef963792958896d656849d23c211e2672b6cb03485732e3fb25"} Sep 11 05:27:20.239326 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:20.239290 2577 generic.go:356] "Generic (PLEG): container finished" podID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerID="f0c2191aadca2ef963792958896d656849d23c211e2672b6cb03485732e3fb25" exitCode=0 Sep 11 05:27:20.239715 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:20.239364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerDied","Data":"f0c2191aadca2ef963792958896d656849d23c211e2672b6cb03485732e3fb25"} Sep 11 05:27:21.250689 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:21.250648 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerStarted","Data":"e96448bc3ce40bd63228d14496a395acf79619bd923e78f850d84581c72d498d"} Sep 11 05:27:21.251166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:21.250716 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerStarted","Data":"cf03a61b60f4280e6ceb5bd21fbebf03a34485d54dbbc13bc4a2e257061f9f63"} Sep 11 05:27:21.251166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:21.250736 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerStarted","Data":"4d2afbac432c6ec8bfcc569a79964ab7056b1202e1b3a5f7927d3bb5d21ee710"} Sep 11 05:27:21.251166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:21.250757 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerStarted","Data":"6fd42ee61c8aa35061a53aaa96a41b27f9ee4459260694cd74698a01f36fd498"} Sep 11 05:27:21.279535 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:21.279452 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podStartSLOduration=13.882201797 podStartE2EDuration="39.279428937s" totalImagesPullingTime="25.39722714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:27:21.277977352 +0000 UTC m=+414.387715533" watchObservedRunningTime="2026-09-11 05:27:21.279428937 +0000 UTC m=+414.389167105" Sep 11 05:27:22.210279 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:22.210145 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:22.210279 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:22.210207 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:22.210279 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:22.210225 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:22.210279 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:22.210244 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:22.212261 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:22.212209 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 11 05:27:22.212261 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:22.212241 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 11 05:27:22.212467 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:22.212232 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 11 05:27:22.212467 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:22.212293 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 05:27:23.209401 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:23.209356 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:23.209914 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:23.209431 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:23.209914 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:23.209451 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:23.209914 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:23.209469 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:25.210430 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:25.210380 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 11 05:27:25.210430 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:25.210415 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 05:27:25.210998 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:25.210388 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 11 05:27:25.210998 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:25.210968 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 11 05:27:28.209900 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:28.209812 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 11 05:27:28.209900 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:28.209850 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 05:27:28.210794 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:28.210769 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:28.210975 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:28.210961 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:28.211637 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:28.211614 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:28.211746 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:28.211684 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:31.210441 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:31.210402 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:31.211137 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:31.211119 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:31.215799 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:31.215778 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:27:31.310226 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:27:31.310197 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:28:33.110033 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.109999 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8"] Sep 11 05:28:33.110596 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.110247 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="registry" containerID="cri-o://6fd42ee61c8aa35061a53aaa96a41b27f9ee4459260694cd74698a01f36fd498" gracePeriod=30 Sep 11 05:28:33.110596 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.110281 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" containerID="cri-o://e96448bc3ce40bd63228d14496a395acf79619bd923e78f850d84581c72d498d" gracePeriod=30 Sep 11 05:28:33.110596 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.110286 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="offline" containerID="cri-o://cf03a61b60f4280e6ceb5bd21fbebf03a34485d54dbbc13bc4a2e257061f9f63" gracePeriod=30 Sep 11 05:28:33.110596 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.110295 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" containerID="cri-o://4d2afbac432c6ec8bfcc569a79964ab7056b1202e1b3a5f7927d3bb5d21ee710" gracePeriod=30 Sep 11 05:28:33.486860 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.486776 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj"] Sep 11 05:28:33.491380 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.491360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.494020 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.493997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-kdbxl\"" Sep 11 05:28:33.506021 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.505999 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj"] Sep 11 05:28:33.610305 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.610259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-ui-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.610501 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.610325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76lq9\" (UniqueName: \"kubernetes.io/projected/c6a46263-58c8-4a43-a782-b4a41559131a-kube-api-access-76lq9\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.610501 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.610413 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-registry-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.610657 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.610499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a46263-58c8-4a43-a782-b4a41559131a-feast-data\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.610657 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.610563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-offline-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.610657 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.610605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.711840 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.711815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-ui-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.711931 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.711854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76lq9\" (UniqueName: \"kubernetes.io/projected/c6a46263-58c8-4a43-a782-b4a41559131a-kube-api-access-76lq9\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.711971 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.711928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-registry-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.712012 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.712002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a46263-58c8-4a43-a782-b4a41559131a-feast-data\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.712061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.712046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-offline-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.712099 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.712087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.712190 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:28:33.712167 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 11 05:28:33.712269 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:28:33.712182 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 05:28:33.712325 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:28:33.712258 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-offline-tls podName:c6a46263-58c8-4a43-a782-b4a41559131a nodeName:}" failed. No retries permitted until 2026-09-11 05:28:34.21223661 +0000 UTC m=+487.321974760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-offline-tls") pod "feast-simple-feast-setup-585fd8b5bc-kv4sj" (UID: "c6a46263-58c8-4a43-a782-b4a41559131a") : secret "feast-simple-feast-setup-offline-tls" not found Sep 11 05:28:33.712325 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:28:33.712318 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls podName:c6a46263-58c8-4a43-a782-b4a41559131a nodeName:}" failed. No retries permitted until 2026-09-11 05:28:34.212305815 +0000 UTC m=+487.322043962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls") pod "feast-simple-feast-setup-585fd8b5bc-kv4sj" (UID: "c6a46263-58c8-4a43-a782-b4a41559131a") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 05:28:33.712495 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.712469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a46263-58c8-4a43-a782-b4a41559131a-feast-data\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.714437 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.714414 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-ui-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.714527 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.714453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-registry-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:33.720383 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:33.720355 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76lq9\" (UniqueName: \"kubernetes.io/projected/c6a46263-58c8-4a43-a782-b4a41559131a-kube-api-access-76lq9\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:34.217007 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:34.216926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-offline-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:34.217007 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:34.216967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:34.217380 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:28:34.217066 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 05:28:34.217380 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:28:34.217118 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls podName:c6a46263-58c8-4a43-a782-b4a41559131a nodeName:}" failed. No retries permitted until 2026-09-11 05:28:35.217103423 +0000 UTC m=+488.326841568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls") pod "feast-simple-feast-setup-585fd8b5bc-kv4sj" (UID: "c6a46263-58c8-4a43-a782-b4a41559131a") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 05:28:34.219259 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:34.219239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-offline-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:34.569598 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:34.569566 2577 generic.go:356] "Generic (PLEG): container finished" podID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerID="e96448bc3ce40bd63228d14496a395acf79619bd923e78f850d84581c72d498d" exitCode=0 Sep 11 05:28:34.569598 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:34.569589 2577 generic.go:356] "Generic (PLEG): container finished" podID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerID="4d2afbac432c6ec8bfcc569a79964ab7056b1202e1b3a5f7927d3bb5d21ee710" exitCode=0 Sep 11 05:28:34.569797 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:34.569622 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerDied","Data":"e96448bc3ce40bd63228d14496a395acf79619bd923e78f850d84581c72d498d"} Sep 11 05:28:34.569797 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:34.569665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerDied","Data":"4d2afbac432c6ec8bfcc569a79964ab7056b1202e1b3a5f7927d3bb5d21ee710"} Sep 11 05:28:35.225596 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:35.225546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:35.228014 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:35.227992 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls\") pod \"feast-simple-feast-setup-585fd8b5bc-kv4sj\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:35.301157 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:35.301127 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:35.579127 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:35.579099 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj"] Sep 11 05:28:35.579505 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:28:35.579482 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6a46263_58c8_4a43_a782_b4a41559131a.slice/crio-5784cfebedc6b3c8fa97fe594e9df9524db12b16d125e1fcfbd8f715898ef50d WatchSource:0}: Error finding container 5784cfebedc6b3c8fa97fe594e9df9524db12b16d125e1fcfbd8f715898ef50d: Status 404 returned error can't find the container with id 5784cfebedc6b3c8fa97fe594e9df9524db12b16d125e1fcfbd8f715898ef50d Sep 11 05:28:36.579186 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:36.579149 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerStarted","Data":"3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307"} Sep 11 05:28:36.579186 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:36.579192 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerStarted","Data":"5784cfebedc6b3c8fa97fe594e9df9524db12b16d125e1fcfbd8f715898ef50d"} Sep 11 05:28:40.597201 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:40.597161 2577 generic.go:356] "Generic (PLEG): container finished" podID="c6a46263-58c8-4a43-a782-b4a41559131a" containerID="3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307" exitCode=0 Sep 11 05:28:40.597650 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:40.597235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerDied","Data":"3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307"} Sep 11 05:28:41.211032 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:41.210991 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 11 05:28:41.308073 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:41.308026 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 05:28:41.603589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:41.603541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerStarted","Data":"92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3"} Sep 11 05:28:46.625449 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:46.625369 2577 generic.go:356] "Generic (PLEG): container finished" podID="c6a46263-58c8-4a43-a782-b4a41559131a" containerID="92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3" exitCode=0 Sep 11 05:28:46.625449 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:46.625433 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerDied","Data":"92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3"} Sep 11 05:28:47.635731 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:47.635691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerStarted","Data":"3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6"} Sep 11 05:28:47.636201 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:47.635752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerStarted","Data":"2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2"} Sep 11 05:28:47.636201 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:47.635773 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerStarted","Data":"4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4"} Sep 11 05:28:47.636201 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:47.635791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerStarted","Data":"eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a"} Sep 11 05:28:47.662850 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:47.662778 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podStartSLOduration=14.662757973 podStartE2EDuration="14.662757973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:28:47.660842703 +0000 UTC m=+500.770580869" watchObservedRunningTime="2026-09-11 05:28:47.662757973 +0000 UTC m=+500.772496142" Sep 11 05:28:50.302187 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:50.302144 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:50.302809 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:50.302777 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:50.302809 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:50.302810 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:50.302991 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:50.302831 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:50.304494 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:50.304345 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Sep 11 05:28:50.304494 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:50.304391 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 11 05:28:50.304494 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:50.304355 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Sep 11 05:28:50.304494 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:50.304389 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 11 05:28:51.211749 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:51.211702 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 11 05:28:51.308133 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:51.308091 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 05:28:53.302063 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:53.302019 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Sep 11 05:28:53.302565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:53.302114 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 11 05:28:53.302565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:53.302019 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 11 05:28:53.302565 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:53.302019 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Sep 11 05:28:55.301266 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:55.301224 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:55.301668 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:55.301284 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:55.301668 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:55.301297 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:55.301668 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:55.301307 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:56.302470 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:56.302441 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:56.302901 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:56.302580 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:56.302965 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:56.302904 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:56.303438 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:56.303415 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:56.303540 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:56.303474 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:56.303540 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:56.303505 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:56.307928 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:56.307910 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:28:56.691275 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:28:56.691197 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:29:01.211118 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:01.211062 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 11 05:29:01.211519 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:01.211214 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:29:01.307840 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:01.307802 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 05:29:01.308014 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:01.307945 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:29:03.718968 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.718942 2577 generic.go:356] "Generic (PLEG): container finished" podID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerID="cf03a61b60f4280e6ceb5bd21fbebf03a34485d54dbbc13bc4a2e257061f9f63" exitCode=137 Sep 11 05:29:03.718968 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.718961 2577 generic.go:356] "Generic (PLEG): container finished" podID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerID="6fd42ee61c8aa35061a53aaa96a41b27f9ee4459260694cd74698a01f36fd498" exitCode=137 Sep 11 05:29:03.719329 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.719020 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerDied","Data":"cf03a61b60f4280e6ceb5bd21fbebf03a34485d54dbbc13bc4a2e257061f9f63"} Sep 11 05:29:03.719329 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.719069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerDied","Data":"6fd42ee61c8aa35061a53aaa96a41b27f9ee4459260694cd74698a01f36fd498"} Sep 11 05:29:03.756042 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.756022 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:29:03.885097 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.885017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-offline-tls\") pod \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " Sep 11 05:29:03.885097 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.885069 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-feast-data\") pod \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " Sep 11 05:29:03.885097 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.885090 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gq22\" (UniqueName: \"kubernetes.io/projected/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-kube-api-access-8gq22\") pod \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " Sep 11 05:29:03.885328 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.885162 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-online-tls\") pod \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " Sep 11 05:29:03.885328 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.885184 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-ui-tls\") pod \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " Sep 11 05:29:03.885328 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.885207 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-registry-tls\") pod \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\" (UID: \"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8\") " Sep 11 05:29:03.885634 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.885603 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-feast-data" (OuterVolumeSpecName: "feast-data") pod "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" (UID: "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:29:03.887436 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.887410 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" (UID: "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:29:03.887536 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.887414 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-kube-api-access-8gq22" (OuterVolumeSpecName: "kube-api-access-8gq22") pod "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" (UID: "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8"). InnerVolumeSpecName "kube-api-access-8gq22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:29:03.887536 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.887458 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" (UID: "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:29:03.887676 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.887634 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" (UID: "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:29:03.887676 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.887659 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-online-tls" (OuterVolumeSpecName: "online-tls") pod "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" (UID: "c47cdc32-592c-4cd1-9ab8-5d04e72a99e8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:29:03.986654 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.986627 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-feast-data\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:29:03.986654 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.986649 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gq22\" (UniqueName: \"kubernetes.io/projected/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-kube-api-access-8gq22\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:29:03.986654 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.986659 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-online-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:29:03.986835 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.986667 2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-ui-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:29:03.986835 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.986675 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-registry-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:29:03.986835 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:03.986683 2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8-offline-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:29:04.726169 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.726135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" event={"ID":"c47cdc32-592c-4cd1-9ab8-5d04e72a99e8","Type":"ContainerDied","Data":"d33b3cf959d21331d1afead34c46433b213a80fa671d93badac70f08fee77ab0"} Sep 11 05:29:04.726169 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.726158 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8" Sep 11 05:29:04.726699 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.726196 2577 scope.go:117] "RemoveContainer" containerID="e96448bc3ce40bd63228d14496a395acf79619bd923e78f850d84581c72d498d" Sep 11 05:29:04.736278 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.736250 2577 scope.go:117] "RemoveContainer" containerID="cf03a61b60f4280e6ceb5bd21fbebf03a34485d54dbbc13bc4a2e257061f9f63" Sep 11 05:29:04.745721 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.745702 2577 scope.go:117] "RemoveContainer" containerID="4d2afbac432c6ec8bfcc569a79964ab7056b1202e1b3a5f7927d3bb5d21ee710" Sep 11 05:29:04.752973 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.752955 2577 scope.go:117] "RemoveContainer" containerID="6fd42ee61c8aa35061a53aaa96a41b27f9ee4459260694cd74698a01f36fd498" Sep 11 05:29:04.756510 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.756490 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8"] Sep 11 05:29:04.761263 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.761245 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-vdxz8"] Sep 11 05:29:04.761471 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.761456 2577 scope.go:117] "RemoveContainer" containerID="f0c2191aadca2ef963792958896d656849d23c211e2672b6cb03485732e3fb25" Sep 11 05:29:04.768592 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:04.768578 2577 scope.go:117] "RemoveContainer" containerID="6b4507e084e1e7cc97139abef5ffcb148b6f10b028937ff0551a999015a37e50" Sep 11 05:29:05.355617 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:05.355589 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" path="/var/lib/kubelet/pods/c47cdc32-592c-4cd1-9ab8-5d04e72a99e8/volumes" Sep 11 05:29:58.500535 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500494 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt"] Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500902 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500916 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500926 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500931 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500946 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="feast-apply" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500952 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="feast-apply" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500967 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="feast-init" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500973 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="feast-init" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500978 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="registry" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500983 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="registry" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500990 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="offline" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.500995 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="offline" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.501044 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="ui" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.501050 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="registry" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.501059 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="online" Sep 11 05:29:58.501061 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.501066 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47cdc32-592c-4cd1-9ab8-5d04e72a99e8" containerName="offline" Sep 11 05:29:58.504422 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.504405 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.508613 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.508590 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 11 05:29:58.508748 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.508706 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 11 05:29:58.508748 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.508726 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 11 05:29:58.508874 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.508760 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 11 05:29:58.508874 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.508764 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 11 05:29:58.509254 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.509235 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-hv5rj\"" Sep 11 05:29:58.509336 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.509238 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 11 05:29:58.509336 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.509237 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 11 05:29:58.518227 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.518203 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt"] Sep 11 05:29:58.631836 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.631804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e86604d0-d128-4acf-9771-ea648b3de435-registry-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.631836 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.631847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-offline-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.632064 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.631868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-online-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.632064 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.631984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx99t\" (UniqueName: \"kubernetes.io/projected/e86604d0-d128-4acf-9771-ea648b3de435-kube-api-access-kx99t\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.632064 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.632015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-ui-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.632064 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.632034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e86604d0-d128-4acf-9771-ea648b3de435-feast-data\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.732710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.732679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-offline-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.732710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.732712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-online-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.732934 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.732761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx99t\" (UniqueName: \"kubernetes.io/projected/e86604d0-d128-4acf-9771-ea648b3de435-kube-api-access-kx99t\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.732934 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.732780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-ui-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.732934 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.732795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e86604d0-d128-4acf-9771-ea648b3de435-feast-data\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.732934 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.732825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e86604d0-d128-4acf-9771-ea648b3de435-registry-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.732934 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:29:58.732914 2577 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 11 05:29:58.733216 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:29:58.732977 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-online-tls podName:e86604d0-d128-4acf-9771-ea648b3de435 nodeName:}" failed. No retries permitted until 2026-09-11 05:29:59.232960141 +0000 UTC m=+572.342698288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-online-tls") pod "feast-simple-feast-remote-setup-87cfd7d7-rn6tt" (UID: "e86604d0-d128-4acf-9771-ea648b3de435") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 11 05:29:58.733294 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.733273 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e86604d0-d128-4acf-9771-ea648b3de435-feast-data\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.733529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.733514 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e86604d0-d128-4acf-9771-ea648b3de435-registry-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.735156 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.735136 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-ui-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.735246 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.735213 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-offline-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:58.753859 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:58.753802 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx99t\" (UniqueName: \"kubernetes.io/projected/e86604d0-d128-4acf-9771-ea648b3de435-kube-api-access-kx99t\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:59.237028 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:59.236920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-online-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:59.239276 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:59.239255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-online-tls\") pod \"feast-simple-feast-remote-setup-87cfd7d7-rn6tt\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:59.413830 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:59.413794 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:29:59.695978 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:59.695842 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt"] Sep 11 05:29:59.698657 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:29:59.698628 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode86604d0_d128_4acf_9771_ea648b3de435.slice/crio-93aae4313e26cf6a751b3a804e897be8d3722e0d2e8b0c22c8f448fb1ab52b81 WatchSource:0}: Error finding container 93aae4313e26cf6a751b3a804e897be8d3722e0d2e8b0c22c8f448fb1ab52b81: Status 404 returned error can't find the container with id 93aae4313e26cf6a751b3a804e897be8d3722e0d2e8b0c22c8f448fb1ab52b81 Sep 11 05:29:59.988591 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:59.987663 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerStarted","Data":"23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64"} Sep 11 05:29:59.988591 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:29:59.987721 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerStarted","Data":"93aae4313e26cf6a751b3a804e897be8d3722e0d2e8b0c22c8f448fb1ab52b81"} Sep 11 05:30:05.012052 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:05.012028 2577 generic.go:356] "Generic (PLEG): container finished" podID="e86604d0-d128-4acf-9771-ea648b3de435" containerID="23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64" exitCode=0 Sep 11 05:30:05.012421 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:05.012097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerDied","Data":"23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64"} Sep 11 05:30:06.018335 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:06.018299 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerStarted","Data":"b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe"} Sep 11 05:30:26.101759 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:26.101732 2577 generic.go:356] "Generic (PLEG): container finished" podID="e86604d0-d128-4acf-9771-ea648b3de435" containerID="b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe" exitCode=0 Sep 11 05:30:26.102057 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:26.101769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerDied","Data":"b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe"} Sep 11 05:30:27.111432 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:27.111398 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerStarted","Data":"12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a"} Sep 11 05:30:27.111895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:27.111451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerStarted","Data":"c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e"} Sep 11 05:30:27.111895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:27.111468 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerStarted","Data":"68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a"} Sep 11 05:30:27.136121 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:27.135957 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podStartSLOduration=29.135935957 podStartE2EDuration="29.135935957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:30:27.13537269 +0000 UTC m=+600.245110855" watchObservedRunningTime="2026-09-11 05:30:27.135935957 +0000 UTC m=+600.245674123" Sep 11 05:30:29.414245 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:29.414204 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:29.414791 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:29.414267 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:29.414791 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:29.414287 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:29.414791 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:29.414304 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:29.414791 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:29.414320 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:29.414791 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:29.414336 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:29.415710 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:29.415679 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 11 05:30:29.415846 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:29.415706 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 11 05:30:29.415846 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:29.415717 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 11 05:30:32.414915 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:32.414872 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 11 05:30:32.414915 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:32.414901 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 11 05:30:32.415444 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:32.414871 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 11 05:30:35.414924 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:35.414890 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:35.415337 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:35.415120 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:35.415597 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:35.415581 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:35.415670 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:35.415654 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:35.437469 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:35.437450 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:30:36.178303 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:30:36.178276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:31:38.810679 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:38.810646 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt"] Sep 11 05:31:38.811204 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:38.810883 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="online" containerID="cri-o://68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a" gracePeriod=30 Sep 11 05:31:38.811204 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:38.810941 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="ui" containerID="cri-o://12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a" gracePeriod=30 Sep 11 05:31:38.811204 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:38.810946 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="offline" containerID="cri-o://c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e" gracePeriod=30 Sep 11 05:31:38.921166 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:38.921131 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj"] Sep 11 05:31:38.921377 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:38.921353 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="registry" containerID="cri-o://eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a" gracePeriod=30 Sep 11 05:31:38.921449 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:38.921388 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="ui" containerID="cri-o://3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6" gracePeriod=30 Sep 11 05:31:38.921449 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:38.921413 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="online" containerID="cri-o://4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4" gracePeriod=30 Sep 11 05:31:38.921572 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:38.921402 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="offline" containerID="cri-o://2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2" gracePeriod=30 Sep 11 05:31:40.448242 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:40.448212 2577 generic.go:356] "Generic (PLEG): container finished" podID="c6a46263-58c8-4a43-a782-b4a41559131a" containerID="3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6" exitCode=0 Sep 11 05:31:40.448242 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:40.448236 2577 generic.go:356] "Generic (PLEG): container finished" podID="c6a46263-58c8-4a43-a782-b4a41559131a" containerID="4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4" exitCode=0 Sep 11 05:31:40.448669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:40.448280 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerDied","Data":"3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6"} Sep 11 05:31:40.448669 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:40.448320 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerDied","Data":"4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4"} Sep 11 05:31:40.450882 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:40.450863 2577 generic.go:356] "Generic (PLEG): container finished" podID="e86604d0-d128-4acf-9771-ea648b3de435" containerID="12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a" exitCode=0 Sep 11 05:31:40.450882 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:40.450880 2577 generic.go:356] "Generic (PLEG): container finished" podID="e86604d0-d128-4acf-9771-ea648b3de435" containerID="68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a" exitCode=0 Sep 11 05:31:40.451011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:40.450897 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerDied","Data":"12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a"} Sep 11 05:31:40.451011 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:40.450920 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerDied","Data":"68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a"} Sep 11 05:31:45.415488 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:45.415442 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 11 05:31:46.160765 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:46.160721 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 11 05:31:46.303676 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:46.303628 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 11 05:31:46.688612 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:46.688570 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 11 05:31:55.416046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:55.416001 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 11 05:31:56.161255 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:56.161211 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 11 05:31:56.303540 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:56.303492 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 11 05:31:56.688400 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:31:56.688364 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 11 05:32:05.416294 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:05.416251 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 11 05:32:05.416778 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:05.416374 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:32:06.160933 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:06.160883 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 11 05:32:06.161136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:06.161021 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:32:06.303519 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:06.303463 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 11 05:32:06.303757 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:06.303643 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:32:06.688220 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:06.688174 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 11 05:32:06.688591 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:06.688324 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:32:09.466385 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.466355 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:32:09.562275 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.562251 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:32:09.575515 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.575489 2577 generic.go:356] "Generic (PLEG): container finished" podID="e86604d0-d128-4acf-9771-ea648b3de435" containerID="c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e" exitCode=137 Sep 11 05:32:09.575668 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.575539 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" Sep 11 05:32:09.575668 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.575578 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerDied","Data":"c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e"} Sep 11 05:32:09.575668 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.575618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt" event={"ID":"e86604d0-d128-4acf-9771-ea648b3de435","Type":"ContainerDied","Data":"93aae4313e26cf6a751b3a804e897be8d3722e0d2e8b0c22c8f448fb1ab52b81"} Sep 11 05:32:09.575795 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.575642 2577 scope.go:117] "RemoveContainer" containerID="12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a" Sep 11 05:32:09.578517 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.578470 2577 generic.go:356] "Generic (PLEG): container finished" podID="c6a46263-58c8-4a43-a782-b4a41559131a" containerID="2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2" exitCode=137 Sep 11 05:32:09.578517 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.578487 2577 generic.go:356] "Generic (PLEG): container finished" podID="c6a46263-58c8-4a43-a782-b4a41559131a" containerID="eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a" exitCode=137 Sep 11 05:32:09.578517 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.578505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerDied","Data":"2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2"} Sep 11 05:32:09.578700 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.578534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerDied","Data":"eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a"} Sep 11 05:32:09.578700 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.578568 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" Sep 11 05:32:09.578700 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.578569 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj" event={"ID":"c6a46263-58c8-4a43-a782-b4a41559131a","Type":"ContainerDied","Data":"5784cfebedc6b3c8fa97fe594e9df9524db12b16d125e1fcfbd8f715898ef50d"} Sep 11 05:32:09.584290 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.584270 2577 scope.go:117] "RemoveContainer" containerID="c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e" Sep 11 05:32:09.584831 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.584792 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-ui-tls\") pod \"e86604d0-d128-4acf-9771-ea648b3de435\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " Sep 11 05:32:09.584938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.584832 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx99t\" (UniqueName: \"kubernetes.io/projected/e86604d0-d128-4acf-9771-ea648b3de435-kube-api-access-kx99t\") pod \"e86604d0-d128-4acf-9771-ea648b3de435\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " Sep 11 05:32:09.584938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.584858 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e86604d0-d128-4acf-9771-ea648b3de435-feast-data\") pod \"e86604d0-d128-4acf-9771-ea648b3de435\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " Sep 11 05:32:09.584938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.584885 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-online-tls\") pod \"e86604d0-d128-4acf-9771-ea648b3de435\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " Sep 11 05:32:09.584938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.584913 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e86604d0-d128-4acf-9771-ea648b3de435-registry-tls\") pod \"e86604d0-d128-4acf-9771-ea648b3de435\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " Sep 11 05:32:09.584938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.584935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-offline-tls\") pod \"e86604d0-d128-4acf-9771-ea648b3de435\" (UID: \"e86604d0-d128-4acf-9771-ea648b3de435\") " Sep 11 05:32:09.585670 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.585576 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86604d0-d128-4acf-9771-ea648b3de435-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e86604d0-d128-4acf-9771-ea648b3de435" (UID: "e86604d0-d128-4acf-9771-ea648b3de435"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 05:32:09.585897 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.585871 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86604d0-d128-4acf-9771-ea648b3de435-feast-data" (OuterVolumeSpecName: "feast-data") pod "e86604d0-d128-4acf-9771-ea648b3de435" (UID: "e86604d0-d128-4acf-9771-ea648b3de435"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:32:09.587753 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.587728 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e86604d0-d128-4acf-9771-ea648b3de435" (UID: "e86604d0-d128-4acf-9771-ea648b3de435"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:32:09.587878 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.587763 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86604d0-d128-4acf-9771-ea648b3de435-kube-api-access-kx99t" (OuterVolumeSpecName: "kube-api-access-kx99t") pod "e86604d0-d128-4acf-9771-ea648b3de435" (UID: "e86604d0-d128-4acf-9771-ea648b3de435"). InnerVolumeSpecName "kube-api-access-kx99t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:32:09.588041 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.588019 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-online-tls" (OuterVolumeSpecName: "online-tls") pod "e86604d0-d128-4acf-9771-ea648b3de435" (UID: "e86604d0-d128-4acf-9771-ea648b3de435"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:32:09.588307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.588289 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e86604d0-d128-4acf-9771-ea648b3de435" (UID: "e86604d0-d128-4acf-9771-ea648b3de435"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:32:09.594692 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.594675 2577 scope.go:117] "RemoveContainer" containerID="68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a" Sep 11 05:32:09.603630 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.603590 2577 scope.go:117] "RemoveContainer" containerID="b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe" Sep 11 05:32:09.610355 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.610341 2577 scope.go:117] "RemoveContainer" containerID="23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64" Sep 11 05:32:09.616985 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.616967 2577 scope.go:117] "RemoveContainer" containerID="12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a" Sep 11 05:32:09.617262 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.617239 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a\": container with ID starting with 12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a not found: ID does not exist" containerID="12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a" Sep 11 05:32:09.617300 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.617271 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a"} err="failed to get container status \"12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a\": rpc error: code = NotFound desc = could not find container \"12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a\": container with ID starting with 12ab6e4e7e8f7b263edbb1bf3a2293f1583be9a7797f191890632a297b27259a not found: ID does not exist" Sep 11 05:32:09.617300 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.617288 2577 scope.go:117] "RemoveContainer" containerID="c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e" Sep 11 05:32:09.617539 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.617517 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e\": container with ID starting with c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e not found: ID does not exist" containerID="c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e" Sep 11 05:32:09.617598 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.617538 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e"} err="failed to get container status \"c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e\": rpc error: code = NotFound desc = could not find container \"c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e\": container with ID starting with c4754b13e9259f0126574a6fa2300b51d9c753e8e5dcf9034b478c81dcc61c2e not found: ID does not exist" Sep 11 05:32:09.617598 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.617566 2577 scope.go:117] "RemoveContainer" containerID="68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a" Sep 11 05:32:09.617779 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.617763 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a\": container with ID starting with 68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a not found: ID does not exist" containerID="68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a" Sep 11 05:32:09.617827 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.617784 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a"} err="failed to get container status \"68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a\": rpc error: code = NotFound desc = could not find container \"68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a\": container with ID starting with 68b5a85ef6b6feafcbe9062a65d2bb620d9d782df83ef9f7000432aad30d489a not found: ID does not exist" Sep 11 05:32:09.617827 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.617798 2577 scope.go:117] "RemoveContainer" containerID="b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe" Sep 11 05:32:09.618015 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.617999 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe\": container with ID starting with b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe not found: ID does not exist" containerID="b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe" Sep 11 05:32:09.618048 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.618021 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe"} err="failed to get container status \"b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe\": rpc error: code = NotFound desc = could not find container \"b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe\": container with ID starting with b10d4a0ab35049e995f2e957d196ce311103c588b58329e1cb7280c825f601fe not found: ID does not exist" Sep 11 05:32:09.618048 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.618044 2577 scope.go:117] "RemoveContainer" containerID="23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64" Sep 11 05:32:09.618282 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.618268 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64\": container with ID starting with 23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64 not found: ID does not exist" containerID="23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64" Sep 11 05:32:09.618320 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.618284 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64"} err="failed to get container status \"23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64\": rpc error: code = NotFound desc = could not find container \"23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64\": container with ID starting with 23d628c427ae4aac4dece8e9ca1cd167164f60c4dc57d180a96d179315addc64 not found: ID does not exist" Sep 11 05:32:09.618320 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.618296 2577 scope.go:117] "RemoveContainer" containerID="3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6" Sep 11 05:32:09.625012 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.624996 2577 scope.go:117] "RemoveContainer" containerID="2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2" Sep 11 05:32:09.631809 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.631790 2577 scope.go:117] "RemoveContainer" containerID="4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4" Sep 11 05:32:09.638495 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.638477 2577 scope.go:117] "RemoveContainer" containerID="eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a" Sep 11 05:32:09.645373 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.645358 2577 scope.go:117] "RemoveContainer" containerID="92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3" Sep 11 05:32:09.651969 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.651954 2577 scope.go:117] "RemoveContainer" containerID="3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307" Sep 11 05:32:09.660659 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.660572 2577 scope.go:117] "RemoveContainer" containerID="3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6" Sep 11 05:32:09.661112 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.661086 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6\": container with ID starting with 3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6 not found: ID does not exist" containerID="3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6" Sep 11 05:32:09.661164 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.661118 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6"} err="failed to get container status \"3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6\": rpc error: code = NotFound desc = could not find container \"3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6\": container with ID starting with 3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6 not found: ID does not exist" Sep 11 05:32:09.661164 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.661135 2577 scope.go:117] "RemoveContainer" containerID="2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2" Sep 11 05:32:09.661388 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.661371 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2\": container with ID starting with 2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2 not found: ID does not exist" containerID="2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2" Sep 11 05:32:09.661469 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.661396 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2"} err="failed to get container status \"2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2\": rpc error: code = NotFound desc = could not find container \"2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2\": container with ID starting with 2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2 not found: ID does not exist" Sep 11 05:32:09.661469 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.661419 2577 scope.go:117] "RemoveContainer" containerID="4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4" Sep 11 05:32:09.661657 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.661640 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4\": container with ID starting with 4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4 not found: ID does not exist" containerID="4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4" Sep 11 05:32:09.661732 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.661661 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4"} err="failed to get container status \"4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4\": rpc error: code = NotFound desc = could not find container \"4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4\": container with ID starting with 4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4 not found: ID does not exist" Sep 11 05:32:09.661732 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.661675 2577 scope.go:117] "RemoveContainer" containerID="eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a" Sep 11 05:32:09.661885 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.661870 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a\": container with ID starting with eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a not found: ID does not exist" containerID="eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a" Sep 11 05:32:09.661924 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.661887 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a"} err="failed to get container status \"eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a\": rpc error: code = NotFound desc = could not find container \"eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a\": container with ID starting with eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a not found: ID does not exist" Sep 11 05:32:09.661924 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.661899 2577 scope.go:117] "RemoveContainer" containerID="92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3" Sep 11 05:32:09.662147 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.662130 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3\": container with ID starting with 92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3 not found: ID does not exist" containerID="92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3" Sep 11 05:32:09.662212 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.662154 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3"} err="failed to get container status \"92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3\": rpc error: code = NotFound desc = could not find container \"92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3\": container with ID starting with 92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3 not found: ID does not exist" Sep 11 05:32:09.662212 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.662173 2577 scope.go:117] "RemoveContainer" containerID="3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307" Sep 11 05:32:09.662443 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:32:09.662426 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307\": container with ID starting with 3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307 not found: ID does not exist" containerID="3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307" Sep 11 05:32:09.662494 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.662454 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307"} err="failed to get container status \"3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307\": rpc error: code = NotFound desc = could not find container \"3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307\": container with ID starting with 3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307 not found: ID does not exist" Sep 11 05:32:09.662494 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.662469 2577 scope.go:117] "RemoveContainer" containerID="3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6" Sep 11 05:32:09.662732 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.662714 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6"} err="failed to get container status \"3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6\": rpc error: code = NotFound desc = could not find container \"3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6\": container with ID starting with 3844aa2482d4e28f3264c727f308cad4ff105495ef2c1feaf2abdf1622976cd6 not found: ID does not exist" Sep 11 05:32:09.662807 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.662733 2577 scope.go:117] "RemoveContainer" containerID="2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2" Sep 11 05:32:09.662965 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.662949 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2"} err="failed to get container status \"2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2\": rpc error: code = NotFound desc = could not find container \"2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2\": container with ID starting with 2a237bd4841d80b8250da16417b17f7a7cfb045f39bd6f862a6e5d46c920fbe2 not found: ID does not exist" Sep 11 05:32:09.663012 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.662965 2577 scope.go:117] "RemoveContainer" containerID="4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4" Sep 11 05:32:09.663177 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.663158 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4"} err="failed to get container status \"4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4\": rpc error: code = NotFound desc = could not find container \"4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4\": container with ID starting with 4d3f54fc34b9484625d7caf0fd02da8e2a25716708078412c3e304cda7ed65f4 not found: ID does not exist" Sep 11 05:32:09.663221 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.663177 2577 scope.go:117] "RemoveContainer" containerID="eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a" Sep 11 05:32:09.663391 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.663374 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a"} err="failed to get container status \"eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a\": rpc error: code = NotFound desc = could not find container \"eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a\": container with ID starting with eebafec6e75b26397aa562a1f129ea70295458493d607e1b4dc4d5e4da08aa3a not found: ID does not exist" Sep 11 05:32:09.663458 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.663392 2577 scope.go:117] "RemoveContainer" containerID="92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3" Sep 11 05:32:09.663653 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.663634 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3"} err="failed to get container status \"92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3\": rpc error: code = NotFound desc = could not find container \"92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3\": container with ID starting with 92c816599180cb790ad5281f1ae807765cd41168663b9796f7f5387995d562a3 not found: ID does not exist" Sep 11 05:32:09.663706 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.663654 2577 scope.go:117] "RemoveContainer" containerID="3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307" Sep 11 05:32:09.663864 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.663847 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307"} err="failed to get container status \"3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307\": rpc error: code = NotFound desc = could not find container \"3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307\": container with ID starting with 3e3ffe8b4c369715e3d41e2a7292d7493dd7c6e7de352272c1acf30db5463307 not found: ID does not exist" Sep 11 05:32:09.686305 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686284 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a46263-58c8-4a43-a782-b4a41559131a-feast-data\") pod \"c6a46263-58c8-4a43-a782-b4a41559131a\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " Sep 11 05:32:09.686396 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686324 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-offline-tls\") pod \"c6a46263-58c8-4a43-a782-b4a41559131a\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " Sep 11 05:32:09.686396 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686369 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-registry-tls\") pod \"c6a46263-58c8-4a43-a782-b4a41559131a\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " Sep 11 05:32:09.686480 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686396 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76lq9\" (UniqueName: \"kubernetes.io/projected/c6a46263-58c8-4a43-a782-b4a41559131a-kube-api-access-76lq9\") pod \"c6a46263-58c8-4a43-a782-b4a41559131a\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " Sep 11 05:32:09.686480 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls\") pod \"c6a46263-58c8-4a43-a782-b4a41559131a\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " Sep 11 05:32:09.686605 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686514 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-ui-tls\") pod \"c6a46263-58c8-4a43-a782-b4a41559131a\" (UID: \"c6a46263-58c8-4a43-a782-b4a41559131a\") " Sep 11 05:32:09.686858 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686837 2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-ui-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.686951 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686866 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx99t\" (UniqueName: \"kubernetes.io/projected/e86604d0-d128-4acf-9771-ea648b3de435-kube-api-access-kx99t\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.686951 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686871 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a46263-58c8-4a43-a782-b4a41559131a-feast-data" (OuterVolumeSpecName: "feast-data") pod "c6a46263-58c8-4a43-a782-b4a41559131a" (UID: "c6a46263-58c8-4a43-a782-b4a41559131a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:32:09.686951 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686881 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e86604d0-d128-4acf-9771-ea648b3de435-feast-data\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.686951 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686895 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-online-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.686951 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686907 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e86604d0-d128-4acf-9771-ea648b3de435-registry-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.686951 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.686919 2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e86604d0-d128-4acf-9771-ea648b3de435-offline-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.688531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.688507 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c6a46263-58c8-4a43-a782-b4a41559131a" (UID: "c6a46263-58c8-4a43-a782-b4a41559131a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:32:09.688531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.688522 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c6a46263-58c8-4a43-a782-b4a41559131a" (UID: "c6a46263-58c8-4a43-a782-b4a41559131a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:32:09.688655 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.688513 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a46263-58c8-4a43-a782-b4a41559131a-kube-api-access-76lq9" (OuterVolumeSpecName: "kube-api-access-76lq9") pod "c6a46263-58c8-4a43-a782-b4a41559131a" (UID: "c6a46263-58c8-4a43-a782-b4a41559131a"). InnerVolumeSpecName "kube-api-access-76lq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:32:09.688836 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.688816 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls" (OuterVolumeSpecName: "online-tls") pod "c6a46263-58c8-4a43-a782-b4a41559131a" (UID: "c6a46263-58c8-4a43-a782-b4a41559131a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:32:09.688877 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.688833 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c6a46263-58c8-4a43-a782-b4a41559131a" (UID: "c6a46263-58c8-4a43-a782-b4a41559131a"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:32:09.787663 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.787612 2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-ui-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.787663 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.787657 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a46263-58c8-4a43-a782-b4a41559131a-feast-data\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.787663 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.787672 2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-offline-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.787890 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.787683 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-registry-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.787890 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.787696 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76lq9\" (UniqueName: \"kubernetes.io/projected/c6a46263-58c8-4a43-a782-b4a41559131a-kube-api-access-76lq9\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.787890 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.787707 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6a46263-58c8-4a43-a782-b4a41559131a-online-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:32:09.905430 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.905403 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt"] Sep 11 05:32:09.909045 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.909018 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-87cfd7d7-rn6tt"] Sep 11 05:32:09.922254 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.922230 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj"] Sep 11 05:32:09.926777 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:09.926756 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-585fd8b5bc-kv4sj"] Sep 11 05:32:11.354470 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:11.354414 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" path="/var/lib/kubelet/pods/c6a46263-58c8-4a43-a782-b4a41559131a/volumes" Sep 11 05:32:11.355077 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:11.355062 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e86604d0-d128-4acf-9771-ea648b3de435" path="/var/lib/kubelet/pods/e86604d0-d128-4acf-9771-ea648b3de435/volumes" Sep 11 05:32:21.687368 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687293 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-zkmft"] Sep 11 05:32:21.687719 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687675 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="online" Sep 11 05:32:21.687719 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687689 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="online" Sep 11 05:32:21.687719 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687701 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="feast-apply" Sep 11 05:32:21.687719 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687707 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="feast-apply" Sep 11 05:32:21.687719 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687713 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="feast-init" Sep 11 05:32:21.687719 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687719 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="feast-init" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687728 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="registry" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687733 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="registry" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687744 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="feast-apply" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687749 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="feast-apply" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687758 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="ui" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687765 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="ui" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687774 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="feast-init" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687779 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="feast-init" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687788 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="offline" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687792 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="offline" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687798 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="online" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687803 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="online" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687809 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="offline" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687814 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="offline" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687823 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="ui" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687828 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="ui" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687876 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="ui" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687883 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="offline" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687892 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="ui" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687897 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="registry" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687902 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="offline" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687907 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e86604d0-d128-4acf-9771-ea648b3de435" containerName="online" Sep 11 05:32:21.687905 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.687913 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a46263-58c8-4a43-a782-b4a41559131a" containerName="online" Sep 11 05:32:21.692869 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.692846 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:32:21.695327 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.695304 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-jpmpp\"" Sep 11 05:32:21.695327 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.695317 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 11 05:32:21.695496 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.695317 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 05:32:21.696348 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.696329 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 05:32:21.699981 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.699952 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-zkmft"] Sep 11 05:32:21.715255 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.715229 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-42pfb"] Sep 11 05:32:21.718373 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.718350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-42pfb" Sep 11 05:32:21.726430 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.726401 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-42pfb"] Sep 11 05:32:21.809726 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.809697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/19a2489b-61be-4c91-8856-43e28b79499f-postgresdata\") pod \"postgres-864846d4f5-zkmft\" (UID: \"19a2489b-61be-4c91-8856-43e28b79499f\") " pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:32:21.809890 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.809745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwfjw\" (UniqueName: \"kubernetes.io/projected/19a2489b-61be-4c91-8856-43e28b79499f-kube-api-access-nwfjw\") pod \"postgres-864846d4f5-zkmft\" (UID: \"19a2489b-61be-4c91-8856-43e28b79499f\") " pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:32:21.809890 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.809866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6dl\" (UniqueName: \"kubernetes.io/projected/82512a4d-e727-4c38-a2ac-99bcab5b8d4a-kube-api-access-fj6dl\") pod \"redis-69bbd798b6-42pfb\" (UID: \"82512a4d-e727-4c38-a2ac-99bcab5b8d4a\") " pod="test-ns-feast/redis-69bbd798b6-42pfb" Sep 11 05:32:21.911006 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.910958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwfjw\" (UniqueName: \"kubernetes.io/projected/19a2489b-61be-4c91-8856-43e28b79499f-kube-api-access-nwfjw\") pod \"postgres-864846d4f5-zkmft\" (UID: \"19a2489b-61be-4c91-8856-43e28b79499f\") " pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:32:21.911218 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.911042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6dl\" (UniqueName: \"kubernetes.io/projected/82512a4d-e727-4c38-a2ac-99bcab5b8d4a-kube-api-access-fj6dl\") pod \"redis-69bbd798b6-42pfb\" (UID: \"82512a4d-e727-4c38-a2ac-99bcab5b8d4a\") " pod="test-ns-feast/redis-69bbd798b6-42pfb" Sep 11 05:32:21.911218 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.911073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/19a2489b-61be-4c91-8856-43e28b79499f-postgresdata\") pod \"postgres-864846d4f5-zkmft\" (UID: \"19a2489b-61be-4c91-8856-43e28b79499f\") " pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:32:21.911475 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.911458 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/19a2489b-61be-4c91-8856-43e28b79499f-postgresdata\") pod \"postgres-864846d4f5-zkmft\" (UID: \"19a2489b-61be-4c91-8856-43e28b79499f\") " pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:32:21.920229 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.920195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6dl\" (UniqueName: \"kubernetes.io/projected/82512a4d-e727-4c38-a2ac-99bcab5b8d4a-kube-api-access-fj6dl\") pod \"redis-69bbd798b6-42pfb\" (UID: \"82512a4d-e727-4c38-a2ac-99bcab5b8d4a\") " pod="test-ns-feast/redis-69bbd798b6-42pfb" Sep 11 05:32:21.920340 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:21.920325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwfjw\" (UniqueName: \"kubernetes.io/projected/19a2489b-61be-4c91-8856-43e28b79499f-kube-api-access-nwfjw\") pod \"postgres-864846d4f5-zkmft\" (UID: \"19a2489b-61be-4c91-8856-43e28b79499f\") " pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:32:22.011774 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:22.011749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:32:22.027738 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:22.027716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-42pfb" Sep 11 05:32:22.281273 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:22.281196 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-zkmft"] Sep 11 05:32:22.284406 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:32:22.284378 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19a2489b_61be_4c91_8856_43e28b79499f.slice/crio-985399bb271c9f01d64f7284dc80e114cc1b8d0d41f8f276349a80667c00eb5a WatchSource:0}: Error finding container 985399bb271c9f01d64f7284dc80e114cc1b8d0d41f8f276349a80667c00eb5a: Status 404 returned error can't find the container with id 985399bb271c9f01d64f7284dc80e114cc1b8d0d41f8f276349a80667c00eb5a Sep 11 05:32:22.286338 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:22.286321 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 05:32:22.304251 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:22.304223 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-42pfb"] Sep 11 05:32:22.307253 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:32:22.307223 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82512a4d_e727_4c38_a2ac_99bcab5b8d4a.slice/crio-dd1e50fa18dc3477e408ccfa5fa16a846c7a78a6044efa10408e158994682f2c WatchSource:0}: Error finding container dd1e50fa18dc3477e408ccfa5fa16a846c7a78a6044efa10408e158994682f2c: Status 404 returned error can't find the container with id dd1e50fa18dc3477e408ccfa5fa16a846c7a78a6044efa10408e158994682f2c Sep 11 05:32:22.633862 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:22.633781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-42pfb" event={"ID":"82512a4d-e727-4c38-a2ac-99bcab5b8d4a","Type":"ContainerStarted","Data":"dd1e50fa18dc3477e408ccfa5fa16a846c7a78a6044efa10408e158994682f2c"} Sep 11 05:32:22.634914 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:22.634891 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-zkmft" event={"ID":"19a2489b-61be-4c91-8856-43e28b79499f","Type":"ContainerStarted","Data":"985399bb271c9f01d64f7284dc80e114cc1b8d0d41f8f276349a80667c00eb5a"} Sep 11 05:32:25.649986 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:25.649947 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-42pfb" event={"ID":"82512a4d-e727-4c38-a2ac-99bcab5b8d4a","Type":"ContainerStarted","Data":"e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10"} Sep 11 05:32:25.665028 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:25.664974 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-42pfb" podStartSLOduration=1.861663027 podStartE2EDuration="4.664955677s" totalImagesPullingTime="2.80329265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:32:25.664776989 +0000 UTC m=+718.774515159" watchObservedRunningTime="2026-09-11 05:32:25.664955677 +0000 UTC m=+718.774693844" Sep 11 05:32:27.355792 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:27.355771 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 11 05:32:27.660994 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:27.660915 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-zkmft" event={"ID":"19a2489b-61be-4c91-8856-43e28b79499f","Type":"ContainerStarted","Data":"e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698"} Sep 11 05:32:27.676107 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:27.676055 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-zkmft" podStartSLOduration=1.609623908 podStartE2EDuration="6.676038674s" totalImagesPullingTime="5.066414766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:32:27.675386863 +0000 UTC m=+720.785125029" watchObservedRunningTime="2026-09-11 05:32:27.676038674 +0000 UTC m=+720.785776839" Sep 11 05:32:30.535325 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.535288 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-678455b986-2wdqb"] Sep 11 05:32:30.539163 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.539146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.541749 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.541716 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 11 05:32:30.541749 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.541718 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 11 05:32:30.541954 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.541782 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-rzvpz\"" Sep 11 05:32:30.541954 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.541795 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 11 05:32:30.541954 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.541841 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 11 05:32:30.548782 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.548760 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-678455b986-2wdqb"] Sep 11 05:32:30.690937 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.690905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq55z\" (UniqueName: \"kubernetes.io/projected/09169134-62aa-4472-8702-e12b954b6da6-kube-api-access-bq55z\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.691125 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.690973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/09169134-62aa-4472-8702-e12b954b6da6-feast-data\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.691125 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.691001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-registry-tls\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.691125 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.691045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-offline-tls\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.691125 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.691068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-online-tls\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.792050 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.791966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq55z\" (UniqueName: \"kubernetes.io/projected/09169134-62aa-4472-8702-e12b954b6da6-kube-api-access-bq55z\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.792050 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.792021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/09169134-62aa-4472-8702-e12b954b6da6-feast-data\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.792050 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.792042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-registry-tls\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.792274 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.792071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-offline-tls\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.792274 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.792090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-online-tls\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.792574 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.792527 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/09169134-62aa-4472-8702-e12b954b6da6-feast-data\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.794601 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.794579 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-registry-tls\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.794704 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.794642 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-online-tls\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.794704 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.794682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-offline-tls\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.800217 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.800198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq55z\" (UniqueName: \"kubernetes.io/projected/09169134-62aa-4472-8702-e12b954b6da6-kube-api-access-bq55z\") pod \"feast-credit-scoring-678455b986-2wdqb\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:30.849216 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:30.849176 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:31.127575 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:31.127330 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-678455b986-2wdqb"] Sep 11 05:32:31.129590 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:32:31.129565 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09169134_62aa_4472_8702_e12b954b6da6.slice/crio-4a5477cdfc033d00126f296ce4dc415c5b4f7a8cd8114aa663ff7a0b00d02494 WatchSource:0}: Error finding container 4a5477cdfc033d00126f296ce4dc415c5b4f7a8cd8114aa663ff7a0b00d02494: Status 404 returned error can't find the container with id 4a5477cdfc033d00126f296ce4dc415c5b4f7a8cd8114aa663ff7a0b00d02494 Sep 11 05:32:31.680160 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:31.680125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerStarted","Data":"394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316"} Sep 11 05:32:31.680610 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:31.680183 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerStarted","Data":"4a5477cdfc033d00126f296ce4dc415c5b4f7a8cd8114aa663ff7a0b00d02494"} Sep 11 05:32:32.685672 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:32.685640 2577 generic.go:356] "Generic (PLEG): container finished" podID="09169134-62aa-4472-8702-e12b954b6da6" containerID="394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316" exitCode=0 Sep 11 05:32:32.686048 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:32.685710 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerDied","Data":"394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316"} Sep 11 05:32:33.692234 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:33.692197 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerStarted","Data":"56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25"} Sep 11 05:32:39.719076 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:39.718994 2577 generic.go:356] "Generic (PLEG): container finished" podID="09169134-62aa-4472-8702-e12b954b6da6" containerID="56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25" exitCode=0 Sep 11 05:32:39.719076 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:39.719036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerDied","Data":"56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25"} Sep 11 05:32:40.737192 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:40.737150 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerStarted","Data":"5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb"} Sep 11 05:32:40.737790 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:40.737235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerStarted","Data":"d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019"} Sep 11 05:32:40.737790 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:40.737253 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerStarted","Data":"11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205"} Sep 11 05:32:40.761744 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:40.761694 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podStartSLOduration=10.761680077 podStartE2EDuration="10.761680077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:32:40.760481227 +0000 UTC m=+733.870219393" watchObservedRunningTime="2026-09-11 05:32:40.761680077 +0000 UTC m=+733.871418244" Sep 11 05:32:40.849631 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:40.849595 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:40.849832 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:40.849662 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:40.849832 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:40.849681 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:42.849731 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:42.849685 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:42.850233 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:42.849764 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:42.850233 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:42.849790 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:42.852002 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:42.851703 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 11 05:32:42.852002 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:42.851746 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 05:32:42.852002 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:42.851702 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 11 05:32:45.849916 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:45.849881 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 11 05:32:45.850394 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:45.849928 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 11 05:32:45.850512 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:45.849941 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 05:32:48.850567 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:48.850495 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 11 05:32:48.851605 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:48.851580 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:48.852498 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:48.852476 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:48.856802 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:48.856778 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:49.790864 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:49.790835 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:51.850342 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:51.850303 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 11 05:32:54.850388 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:54.850356 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:32:54.851359 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:32:54.851339 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:33:59.375200 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.375127 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-678455b986-2wdqb"] Sep 11 05:33:59.375768 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.375345 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="registry" containerID="cri-o://11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205" gracePeriod=30 Sep 11 05:33:59.375768 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.375390 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" containerID="cri-o://d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019" gracePeriod=30 Sep 11 05:33:59.375768 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.375399 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="offline" containerID="cri-o://5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb" gracePeriod=30 Sep 11 05:33:59.390940 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.390916 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-zkmft"] Sep 11 05:33:59.391076 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.391060 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-zkmft" podUID="19a2489b-61be-4c91-8856-43e28b79499f" containerName="postgres" containerID="cri-o://e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698" gracePeriod=30 Sep 11 05:33:59.400874 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.400844 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-42pfb"] Sep 11 05:33:59.401053 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.401032 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-42pfb" podUID="82512a4d-e727-4c38-a2ac-99bcab5b8d4a" containerName="redis" containerID="cri-o://e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10" gracePeriod=30 Sep 11 05:33:59.659723 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.659696 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:33:59.662361 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.662342 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-42pfb" Sep 11 05:33:59.768972 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.768942 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwfjw\" (UniqueName: \"kubernetes.io/projected/19a2489b-61be-4c91-8856-43e28b79499f-kube-api-access-nwfjw\") pod \"19a2489b-61be-4c91-8856-43e28b79499f\" (UID: \"19a2489b-61be-4c91-8856-43e28b79499f\") " Sep 11 05:33:59.769158 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.769045 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/19a2489b-61be-4c91-8856-43e28b79499f-postgresdata\") pod \"19a2489b-61be-4c91-8856-43e28b79499f\" (UID: \"19a2489b-61be-4c91-8856-43e28b79499f\") " Sep 11 05:33:59.769158 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.769069 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj6dl\" (UniqueName: \"kubernetes.io/projected/82512a4d-e727-4c38-a2ac-99bcab5b8d4a-kube-api-access-fj6dl\") pod \"82512a4d-e727-4c38-a2ac-99bcab5b8d4a\" (UID: \"82512a4d-e727-4c38-a2ac-99bcab5b8d4a\") " Sep 11 05:33:59.771064 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.771028 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a2489b-61be-4c91-8856-43e28b79499f-kube-api-access-nwfjw" (OuterVolumeSpecName: "kube-api-access-nwfjw") pod "19a2489b-61be-4c91-8856-43e28b79499f" (UID: "19a2489b-61be-4c91-8856-43e28b79499f"). InnerVolumeSpecName "kube-api-access-nwfjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:33:59.771176 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.771079 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82512a4d-e727-4c38-a2ac-99bcab5b8d4a-kube-api-access-fj6dl" (OuterVolumeSpecName: "kube-api-access-fj6dl") pod "82512a4d-e727-4c38-a2ac-99bcab5b8d4a" (UID: "82512a4d-e727-4c38-a2ac-99bcab5b8d4a"). InnerVolumeSpecName "kube-api-access-fj6dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:33:59.788729 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.788692 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 05:33:59.801438 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.801401 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a2489b-61be-4c91-8856-43e28b79499f-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "19a2489b-61be-4c91-8856-43e28b79499f" (UID: "19a2489b-61be-4c91-8856-43e28b79499f"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:33:59.870262 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.870224 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj6dl\" (UniqueName: \"kubernetes.io/projected/82512a4d-e727-4c38-a2ac-99bcab5b8d4a-kube-api-access-fj6dl\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:33:59.870262 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.870252 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwfjw\" (UniqueName: \"kubernetes.io/projected/19a2489b-61be-4c91-8856-43e28b79499f-kube-api-access-nwfjw\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:33:59.870262 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:33:59.870262 2577 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/19a2489b-61be-4c91-8856-43e28b79499f-postgresdata\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:34:00.088931 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.088900 2577 generic.go:356] "Generic (PLEG): container finished" podID="82512a4d-e727-4c38-a2ac-99bcab5b8d4a" containerID="e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10" exitCode=0 Sep 11 05:34:00.089116 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.088944 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-42pfb" Sep 11 05:34:00.089116 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.088978 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-42pfb" event={"ID":"82512a4d-e727-4c38-a2ac-99bcab5b8d4a","Type":"ContainerDied","Data":"e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10"} Sep 11 05:34:00.089116 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.089029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-42pfb" event={"ID":"82512a4d-e727-4c38-a2ac-99bcab5b8d4a","Type":"ContainerDied","Data":"dd1e50fa18dc3477e408ccfa5fa16a846c7a78a6044efa10408e158994682f2c"} Sep 11 05:34:00.089116 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.089053 2577 scope.go:117] "RemoveContainer" containerID="e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10" Sep 11 05:34:00.090362 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.090341 2577 generic.go:356] "Generic (PLEG): container finished" podID="19a2489b-61be-4c91-8856-43e28b79499f" containerID="e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698" exitCode=0 Sep 11 05:34:00.090484 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.090384 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-zkmft" Sep 11 05:34:00.090484 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.090425 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-zkmft" event={"ID":"19a2489b-61be-4c91-8856-43e28b79499f","Type":"ContainerDied","Data":"e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698"} Sep 11 05:34:00.090484 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.090456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-zkmft" event={"ID":"19a2489b-61be-4c91-8856-43e28b79499f","Type":"ContainerDied","Data":"985399bb271c9f01d64f7284dc80e114cc1b8d0d41f8f276349a80667c00eb5a"} Sep 11 05:34:00.100923 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.099784 2577 scope.go:117] "RemoveContainer" containerID="e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10" Sep 11 05:34:00.100923 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:34:00.100165 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10\": container with ID starting with e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10 not found: ID does not exist" containerID="e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10" Sep 11 05:34:00.100923 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.100201 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10"} err="failed to get container status \"e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10\": rpc error: code = NotFound desc = could not find container \"e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10\": container with ID starting with e4d435a1b5c3ae45491ea679b6d1082560d60e50d1b4b7c30f6306b6edc26b10 not found: ID does not exist" Sep 11 05:34:00.100923 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.100222 2577 scope.go:117] "RemoveContainer" containerID="e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698" Sep 11 05:34:00.111114 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.111094 2577 scope.go:117] "RemoveContainer" containerID="e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698" Sep 11 05:34:00.111414 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:34:00.111393 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698\": container with ID starting with e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698 not found: ID does not exist" containerID="e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698" Sep 11 05:34:00.111478 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.111424 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698"} err="failed to get container status \"e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698\": rpc error: code = NotFound desc = could not find container \"e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698\": container with ID starting with e6b4a2fb3b39f3275568446c4e6faa736ce0e5435ecbe4e7ca332acf3fa16698 not found: ID does not exist" Sep 11 05:34:00.118303 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.118284 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-42pfb"] Sep 11 05:34:00.121984 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.121961 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-42pfb"] Sep 11 05:34:00.132672 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.132649 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-zkmft"] Sep 11 05:34:00.136923 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:00.136901 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-zkmft"] Sep 11 05:34:01.098310 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:01.098279 2577 generic.go:356] "Generic (PLEG): container finished" podID="09169134-62aa-4472-8702-e12b954b6da6" containerID="d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019" exitCode=0 Sep 11 05:34:01.098689 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:01.098319 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerDied","Data":"d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019"} Sep 11 05:34:01.356316 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:01.356228 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19a2489b-61be-4c91-8856-43e28b79499f" path="/var/lib/kubelet/pods/19a2489b-61be-4c91-8856-43e28b79499f/volumes" Sep 11 05:34:01.356717 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:01.356690 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82512a4d-e727-4c38-a2ac-99bcab5b8d4a" path="/var/lib/kubelet/pods/82512a4d-e727-4c38-a2ac-99bcab5b8d4a/volumes" Sep 11 05:34:09.787711 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:09.787676 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 05:34:19.788161 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:19.788122 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 05:34:19.788629 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:19.788227 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:34:29.788252 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:29.788217 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 05:34:30.015605 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.015583 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:34:30.026087 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.026067 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-offline-tls\") pod \"09169134-62aa-4472-8702-e12b954b6da6\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " Sep 11 05:34:30.026163 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.026119 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-registry-tls\") pod \"09169134-62aa-4472-8702-e12b954b6da6\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " Sep 11 05:34:30.026163 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.026150 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/09169134-62aa-4472-8702-e12b954b6da6-feast-data\") pod \"09169134-62aa-4472-8702-e12b954b6da6\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " Sep 11 05:34:30.026278 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.026182 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq55z\" (UniqueName: \"kubernetes.io/projected/09169134-62aa-4472-8702-e12b954b6da6-kube-api-access-bq55z\") pod \"09169134-62aa-4472-8702-e12b954b6da6\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " Sep 11 05:34:30.026278 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.026220 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-online-tls\") pod \"09169134-62aa-4472-8702-e12b954b6da6\" (UID: \"09169134-62aa-4472-8702-e12b954b6da6\") " Sep 11 05:34:30.028153 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.028120 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "09169134-62aa-4472-8702-e12b954b6da6" (UID: "09169134-62aa-4472-8702-e12b954b6da6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:34:30.028267 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.028127 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09169134-62aa-4472-8702-e12b954b6da6-kube-api-access-bq55z" (OuterVolumeSpecName: "kube-api-access-bq55z") pod "09169134-62aa-4472-8702-e12b954b6da6" (UID: "09169134-62aa-4472-8702-e12b954b6da6"). InnerVolumeSpecName "kube-api-access-bq55z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:34:30.028498 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.028477 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-online-tls" (OuterVolumeSpecName: "online-tls") pod "09169134-62aa-4472-8702-e12b954b6da6" (UID: "09169134-62aa-4472-8702-e12b954b6da6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:34:30.028601 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.028498 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "09169134-62aa-4472-8702-e12b954b6da6" (UID: "09169134-62aa-4472-8702-e12b954b6da6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:34:30.034529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.034497 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09169134-62aa-4472-8702-e12b954b6da6-feast-data" (OuterVolumeSpecName: "feast-data") pod "09169134-62aa-4472-8702-e12b954b6da6" (UID: "09169134-62aa-4472-8702-e12b954b6da6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:34:30.127634 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.127544 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-online-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:34:30.127634 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.127588 2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-offline-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:34:30.127634 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.127597 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/09169134-62aa-4472-8702-e12b954b6da6-registry-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:34:30.127634 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.127605 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/09169134-62aa-4472-8702-e12b954b6da6-feast-data\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:34:30.127634 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.127613 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq55z\" (UniqueName: \"kubernetes.io/projected/09169134-62aa-4472-8702-e12b954b6da6-kube-api-access-bq55z\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:34:30.219983 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.219952 2577 generic.go:356] "Generic (PLEG): container finished" podID="09169134-62aa-4472-8702-e12b954b6da6" containerID="5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb" exitCode=137 Sep 11 05:34:30.219983 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.219976 2577 generic.go:356] "Generic (PLEG): container finished" podID="09169134-62aa-4472-8702-e12b954b6da6" containerID="11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205" exitCode=137 Sep 11 05:34:30.220257 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.220021 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" Sep 11 05:34:30.220257 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.220033 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerDied","Data":"5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb"} Sep 11 05:34:30.220257 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.220095 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerDied","Data":"11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205"} Sep 11 05:34:30.220257 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.220124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-678455b986-2wdqb" event={"ID":"09169134-62aa-4472-8702-e12b954b6da6","Type":"ContainerDied","Data":"4a5477cdfc033d00126f296ce4dc415c5b4f7a8cd8114aa663ff7a0b00d02494"} Sep 11 05:34:30.220257 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.220155 2577 scope.go:117] "RemoveContainer" containerID="5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb" Sep 11 05:34:30.229146 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.229129 2577 scope.go:117] "RemoveContainer" containerID="d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019" Sep 11 05:34:30.236223 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.236206 2577 scope.go:117] "RemoveContainer" containerID="11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205" Sep 11 05:34:30.241777 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.241752 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-678455b986-2wdqb"] Sep 11 05:34:30.244056 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.244037 2577 scope.go:117] "RemoveContainer" containerID="56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25" Sep 11 05:34:30.245489 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.245470 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-678455b986-2wdqb"] Sep 11 05:34:30.251322 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.251304 2577 scope.go:117] "RemoveContainer" containerID="394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316" Sep 11 05:34:30.258013 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.257997 2577 scope.go:117] "RemoveContainer" containerID="5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb" Sep 11 05:34:30.258254 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:34:30.258238 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb\": container with ID starting with 5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb not found: ID does not exist" containerID="5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb" Sep 11 05:34:30.258306 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.258261 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb"} err="failed to get container status \"5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb\": rpc error: code = NotFound desc = could not find container \"5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb\": container with ID starting with 5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb not found: ID does not exist" Sep 11 05:34:30.258306 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.258282 2577 scope.go:117] "RemoveContainer" containerID="d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019" Sep 11 05:34:30.258521 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:34:30.258501 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019\": container with ID starting with d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019 not found: ID does not exist" containerID="d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019" Sep 11 05:34:30.258580 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.258524 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019"} err="failed to get container status \"d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019\": rpc error: code = NotFound desc = could not find container \"d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019\": container with ID starting with d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019 not found: ID does not exist" Sep 11 05:34:30.258580 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.258536 2577 scope.go:117] "RemoveContainer" containerID="11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205" Sep 11 05:34:30.258763 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:34:30.258748 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205\": container with ID starting with 11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205 not found: ID does not exist" containerID="11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205" Sep 11 05:34:30.258810 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.258766 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205"} err="failed to get container status \"11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205\": rpc error: code = NotFound desc = could not find container \"11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205\": container with ID starting with 11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205 not found: ID does not exist" Sep 11 05:34:30.258810 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.258779 2577 scope.go:117] "RemoveContainer" containerID="56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25" Sep 11 05:34:30.259000 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:34:30.258985 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25\": container with ID starting with 56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25 not found: ID does not exist" containerID="56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25" Sep 11 05:34:30.259047 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259003 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25"} err="failed to get container status \"56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25\": rpc error: code = NotFound desc = could not find container \"56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25\": container with ID starting with 56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25 not found: ID does not exist" Sep 11 05:34:30.259047 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259014 2577 scope.go:117] "RemoveContainer" containerID="394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316" Sep 11 05:34:30.259201 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:34:30.259187 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316\": container with ID starting with 394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316 not found: ID does not exist" containerID="394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316" Sep 11 05:34:30.259242 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259203 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316"} err="failed to get container status \"394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316\": rpc error: code = NotFound desc = could not find container \"394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316\": container with ID starting with 394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316 not found: ID does not exist" Sep 11 05:34:30.259242 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259214 2577 scope.go:117] "RemoveContainer" containerID="5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb" Sep 11 05:34:30.259411 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259397 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb"} err="failed to get container status \"5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb\": rpc error: code = NotFound desc = could not find container \"5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb\": container with ID starting with 5b30e82d668e6939681ac7f678282c5a845e20e1cb8336ae94829b0c188eaefb not found: ID does not exist" Sep 11 05:34:30.259458 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259411 2577 scope.go:117] "RemoveContainer" containerID="d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019" Sep 11 05:34:30.259627 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259606 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019"} err="failed to get container status \"d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019\": rpc error: code = NotFound desc = could not find container \"d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019\": container with ID starting with d491429e35cfd8cfab5cb95bc63919d54e13dc6a974d6586dab2a02d8ade5019 not found: ID does not exist" Sep 11 05:34:30.259712 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259628 2577 scope.go:117] "RemoveContainer" containerID="11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205" Sep 11 05:34:30.259826 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259810 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205"} err="failed to get container status \"11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205\": rpc error: code = NotFound desc = could not find container \"11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205\": container with ID starting with 11cd2cadc769f955b66babfc745e8aa93cbb39712ee322b733d85c3718848205 not found: ID does not exist" Sep 11 05:34:30.259826 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.259826 2577 scope.go:117] "RemoveContainer" containerID="56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25" Sep 11 05:34:30.260087 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.260056 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25"} err="failed to get container status \"56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25\": rpc error: code = NotFound desc = could not find container \"56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25\": container with ID starting with 56ae1e9ff18102a8778efa8ef84b4f35897fca295b79fffeb1ea3697b1a00c25 not found: ID does not exist" Sep 11 05:34:30.260136 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.260089 2577 scope.go:117] "RemoveContainer" containerID="394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316" Sep 11 05:34:30.260313 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:30.260294 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316"} err="failed to get container status \"394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316\": rpc error: code = NotFound desc = could not find container \"394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316\": container with ID starting with 394cad09a896237419234cfdeb46482806f6364c503452b7fed29ff4dd0ee316 not found: ID does not exist" Sep 11 05:34:31.354202 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:34:31.354174 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09169134-62aa-4472-8702-e12b954b6da6" path="/var/lib/kubelet/pods/09169134-62aa-4472-8702-e12b954b6da6/volumes" Sep 11 05:36:36.677587 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.677538 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-75vjp"] Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678044 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82512a4d-e727-4c38-a2ac-99bcab5b8d4a" containerName="redis" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678059 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="82512a4d-e727-4c38-a2ac-99bcab5b8d4a" containerName="redis" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678069 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a2489b-61be-4c91-8856-43e28b79499f" containerName="postgres" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678074 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a2489b-61be-4c91-8856-43e28b79499f" containerName="postgres" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678086 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="feast-init" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678091 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="feast-init" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678099 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="offline" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678104 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="offline" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678112 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="registry" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678117 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="registry" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678124 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678129 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678139 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="feast-apply" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678144 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="feast-apply" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678196 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="82512a4d-e727-4c38-a2ac-99bcab5b8d4a" containerName="redis" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678203 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="online" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678208 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="offline" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678215 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="19a2489b-61be-4c91-8856-43e28b79499f" containerName="postgres" Sep 11 05:36:36.680044 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.678221 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09169134-62aa-4472-8702-e12b954b6da6" containerName="registry" Sep 11 05:36:36.680856 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.680838 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" Sep 11 05:36:36.683351 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.683333 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 11 05:36:36.683452 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.683352 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-bfhtr\"" Sep 11 05:36:36.684520 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.684500 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 11 05:36:36.698265 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.698241 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-75vjp"] Sep 11 05:36:36.716753 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.716731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2swxc\" (UniqueName: \"kubernetes.io/projected/5dfb58ef-9a49-4688-b5ba-393138a1ac4c-kube-api-access-2swxc\") pod \"redis-69bbd798b6-75vjp\" (UID: \"5dfb58ef-9a49-4688-b5ba-393138a1ac4c\") " pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" Sep 11 05:36:36.817566 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.817520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2swxc\" (UniqueName: \"kubernetes.io/projected/5dfb58ef-9a49-4688-b5ba-393138a1ac4c-kube-api-access-2swxc\") pod \"redis-69bbd798b6-75vjp\" (UID: \"5dfb58ef-9a49-4688-b5ba-393138a1ac4c\") " pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" Sep 11 05:36:36.825661 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:36.825635 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2swxc\" (UniqueName: \"kubernetes.io/projected/5dfb58ef-9a49-4688-b5ba-393138a1ac4c-kube-api-access-2swxc\") pod \"redis-69bbd798b6-75vjp\" (UID: \"5dfb58ef-9a49-4688-b5ba-393138a1ac4c\") " pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" Sep 11 05:36:37.003370 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:37.003349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" Sep 11 05:36:37.268615 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:37.268525 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-75vjp"] Sep 11 05:36:37.272199 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:36:37.272173 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dfb58ef_9a49_4688_b5ba_393138a1ac4c.slice/crio-cdb58af37a090d0922fc9c6d2c346e1afcc5d98bd172164bf8d7c6a552e92ca4 WatchSource:0}: Error finding container cdb58af37a090d0922fc9c6d2c346e1afcc5d98bd172164bf8d7c6a552e92ca4: Status 404 returned error can't find the container with id cdb58af37a090d0922fc9c6d2c346e1afcc5d98bd172164bf8d7c6a552e92ca4 Sep 11 05:36:37.730878 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:37.730793 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" event={"ID":"5dfb58ef-9a49-4688-b5ba-393138a1ac4c","Type":"ContainerStarted","Data":"19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd"} Sep 11 05:36:37.730878 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:37.730835 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" event={"ID":"5dfb58ef-9a49-4688-b5ba-393138a1ac4c","Type":"ContainerStarted","Data":"cdb58af37a090d0922fc9c6d2c346e1afcc5d98bd172164bf8d7c6a552e92ca4"} Sep 11 05:36:37.745872 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:37.745824 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" podStartSLOduration=1.745810143 podStartE2EDuration="1.745810143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:36:37.744252208 +0000 UTC m=+970.853990376" watchObservedRunningTime="2026-09-11 05:36:37.745810143 +0000 UTC m=+970.855548308" Sep 11 05:36:42.097651 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.097612 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kzchd"] Sep 11 05:36:42.101092 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.101066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:36:42.103447 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.103425 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 11 05:36:42.106083 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.106061 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kzchd"] Sep 11 05:36:42.157007 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.156981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72p2\" (UniqueName: \"kubernetes.io/projected/d77151ac-a5a5-4502-b7f7-8c7c8f949258-kube-api-access-f72p2\") pod \"postgres-864846d4f5-kzchd\" (UID: \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\") " pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:36:42.157119 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.157052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d77151ac-a5a5-4502-b7f7-8c7c8f949258-postgresdata\") pod \"postgres-864846d4f5-kzchd\" (UID: \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\") " pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:36:42.257868 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.257838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d77151ac-a5a5-4502-b7f7-8c7c8f949258-postgresdata\") pod \"postgres-864846d4f5-kzchd\" (UID: \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\") " pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:36:42.258016 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.257883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72p2\" (UniqueName: \"kubernetes.io/projected/d77151ac-a5a5-4502-b7f7-8c7c8f949258-kube-api-access-f72p2\") pod \"postgres-864846d4f5-kzchd\" (UID: \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\") " pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:36:42.258251 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.258234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d77151ac-a5a5-4502-b7f7-8c7c8f949258-postgresdata\") pod \"postgres-864846d4f5-kzchd\" (UID: \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\") " pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:36:42.266143 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.266113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72p2\" (UniqueName: \"kubernetes.io/projected/d77151ac-a5a5-4502-b7f7-8c7c8f949258-kube-api-access-f72p2\") pod \"postgres-864846d4f5-kzchd\" (UID: \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\") " pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:36:42.411259 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.411178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:36:42.675940 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.675914 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kzchd"] Sep 11 05:36:42.678161 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:36:42.678134 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd77151ac_a5a5_4502_b7f7_8c7c8f949258.slice/crio-bd4e1c1d0fa379ba6c8e1d7e80d2df8c036f5722a26d7245c49139a4e228f09f WatchSource:0}: Error finding container bd4e1c1d0fa379ba6c8e1d7e80d2df8c036f5722a26d7245c49139a4e228f09f: Status 404 returned error can't find the container with id bd4e1c1d0fa379ba6c8e1d7e80d2df8c036f5722a26d7245c49139a4e228f09f Sep 11 05:36:42.751474 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.751448 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" event={"ID":"d77151ac-a5a5-4502-b7f7-8c7c8f949258","Type":"ContainerStarted","Data":"f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466"} Sep 11 05:36:42.751627 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.751483 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" event={"ID":"d77151ac-a5a5-4502-b7f7-8c7c8f949258","Type":"ContainerStarted","Data":"bd4e1c1d0fa379ba6c8e1d7e80d2df8c036f5722a26d7245c49139a4e228f09f"} Sep 11 05:36:42.766441 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:42.766398 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" podStartSLOduration=0.766383425 podStartE2EDuration="766.383425ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:36:42.765086114 +0000 UTC m=+975.874824280" watchObservedRunningTime="2026-09-11 05:36:42.766383425 +0000 UTC m=+975.876121593" Sep 11 05:36:47.673974 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.673942 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp"] Sep 11 05:36:47.677529 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.677512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.679832 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.679809 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 11 05:36:47.679832 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.679819 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-krlzx\"" Sep 11 05:36:47.679982 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.679820 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 11 05:36:47.680091 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.680075 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 11 05:36:47.690812 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.690792 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp"] Sep 11 05:36:47.700875 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.700855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-registry-tls\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.700966 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.700883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/db485323-1f12-416e-b9d2-1225f3c1f738-feast-data\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.700966 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.700910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp9sn\" (UniqueName: \"kubernetes.io/projected/db485323-1f12-416e-b9d2-1225f3c1f738-kube-api-access-pp9sn\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.701083 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.701025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-online-tls\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.802456 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.802430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-registry-tls\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.802597 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.802466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/db485323-1f12-416e-b9d2-1225f3c1f738-feast-data\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.802597 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.802494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp9sn\" (UniqueName: \"kubernetes.io/projected/db485323-1f12-416e-b9d2-1225f3c1f738-kube-api-access-pp9sn\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.802597 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.802535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-online-tls\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.802727 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:36:47.802639 2577 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 11 05:36:47.802727 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:36:47.802698 2577 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 11 05:36:47.802727 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:36:47.802710 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-online-tls podName:db485323-1f12-416e-b9d2-1225f3c1f738 nodeName:}" failed. No retries permitted until 2026-09-11 05:36:48.302690156 +0000 UTC m=+981.412428303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-online-tls") pod "feast-credit-scoring-6f89f7d49-bxkmp" (UID: "db485323-1f12-416e-b9d2-1225f3c1f738") : secret "feast-credit-scoring-online-tls" not found Sep 11 05:36:47.802852 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:36:47.802743 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-registry-tls podName:db485323-1f12-416e-b9d2-1225f3c1f738 nodeName:}" failed. No retries permitted until 2026-09-11 05:36:48.30273191 +0000 UTC m=+981.412470069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-registry-tls") pod "feast-credit-scoring-6f89f7d49-bxkmp" (UID: "db485323-1f12-416e-b9d2-1225f3c1f738") : secret "feast-credit-scoring-registry-tls" not found Sep 11 05:36:47.802852 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.802803 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/db485323-1f12-416e-b9d2-1225f3c1f738-feast-data\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:47.810994 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:47.810970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp9sn\" (UniqueName: \"kubernetes.io/projected/db485323-1f12-416e-b9d2-1225f3c1f738-kube-api-access-pp9sn\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:48.305753 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:48.305718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-registry-tls\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:48.305946 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:48.305788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-online-tls\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:48.308379 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:48.308348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-online-tls\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:48.308489 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:48.308348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-registry-tls\") pod \"feast-credit-scoring-6f89f7d49-bxkmp\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:48.586764 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:48.586668 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:36:48.860674 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:48.860652 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp"] Sep 11 05:36:48.861143 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:36:48.861124 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb485323_1f12_416e_b9d2_1225f3c1f738.slice/crio-67ece111576565b382f34164a064a1e7d9b346577f19d7afe70afbbd87b969f2 WatchSource:0}: Error finding container 67ece111576565b382f34164a064a1e7d9b346577f19d7afe70afbbd87b969f2: Status 404 returned error can't find the container with id 67ece111576565b382f34164a064a1e7d9b346577f19d7afe70afbbd87b969f2 Sep 11 05:36:49.781966 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:49.781883 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerStarted","Data":"596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33"} Sep 11 05:36:49.781966 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:49.781947 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerStarted","Data":"67ece111576565b382f34164a064a1e7d9b346577f19d7afe70afbbd87b969f2"} Sep 11 05:36:50.786953 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:50.786918 2577 generic.go:356] "Generic (PLEG): container finished" podID="db485323-1f12-416e-b9d2-1225f3c1f738" containerID="596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33" exitCode=0 Sep 11 05:36:50.787382 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:50.786990 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerDied","Data":"596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33"} Sep 11 05:36:51.795063 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:51.795026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerStarted","Data":"cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b"} Sep 11 05:36:57.823110 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:57.823078 2577 generic.go:356] "Generic (PLEG): container finished" podID="db485323-1f12-416e-b9d2-1225f3c1f738" containerID="cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b" exitCode=0 Sep 11 05:36:57.823454 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:57.823153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerDied","Data":"cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b"} Sep 11 05:36:58.830957 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:58.830911 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerStarted","Data":"038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364"} Sep 11 05:36:58.831446 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:58.831001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerStarted","Data":"ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6"} Sep 11 05:36:58.851672 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:36:58.851612 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podStartSLOduration=11.85159612 podStartE2EDuration="11.85159612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:36:58.851484587 +0000 UTC m=+991.961222752" watchObservedRunningTime="2026-09-11 05:36:58.85159612 +0000 UTC m=+991.961334287" Sep 11 05:37:00.587246 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:00.587214 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:37:00.587653 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:00.587343 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:37:00.589005 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:00.588971 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 11 05:37:00.589135 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:00.589017 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 05:37:03.587290 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:03.587241 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 05:37:03.587707 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:03.587241 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 11 05:37:06.587357 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:06.587316 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 11 05:37:06.593526 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:06.593497 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:37:06.594012 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:06.593986 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:37:06.596916 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:06.596893 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:37:08.587272 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:08.587238 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:37:09.587811 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:09.587782 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:37:09.881793 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:09.881708 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:37:13.526025 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.525987 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz"] Sep 11 05:37:13.531300 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.531278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.533635 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.533617 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 11 05:37:13.533635 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.533624 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 11 05:37:13.533773 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.533728 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 11 05:37:13.533773 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.533736 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-622bb\"" Sep 11 05:37:13.540171 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.540152 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz"] Sep 11 05:37:13.638206 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.638173 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrl7\" (UniqueName: \"kubernetes.io/projected/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-kube-api-access-lcrl7\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.638376 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.638233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-feast-data\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.638376 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.638312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-registry-tls\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.638376 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.638344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-online-tls\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.739404 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.739376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrl7\" (UniqueName: \"kubernetes.io/projected/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-kube-api-access-lcrl7\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.739589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.739424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-feast-data\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.739589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.739462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-registry-tls\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.739589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.739479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-online-tls\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.739725 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:37:13.739619 2577 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Sep 11 05:37:13.739725 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:37:13.739681 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-online-tls podName:9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b nodeName:}" failed. No retries permitted until 2026-09-11 05:37:14.239662523 +0000 UTC m=+1007.349400685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-online-tls") pod "feast-driver-ranking-674b7f5594-r27wz" (UID: "9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b") : secret "feast-driver-ranking-online-tls" not found Sep 11 05:37:13.739974 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.739950 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-feast-data\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.741895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.741876 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-registry-tls\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:13.748090 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:13.748071 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrl7\" (UniqueName: \"kubernetes.io/projected/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-kube-api-access-lcrl7\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:14.244265 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:14.244227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-online-tls\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:14.246582 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:14.246560 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-online-tls\") pod \"feast-driver-ranking-674b7f5594-r27wz\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:14.441323 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:14.441278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:14.714612 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:14.714586 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz"] Sep 11 05:37:14.715541 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:37:14.715518 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1cd05a_b17a_4fe3_bf37_d386ea2ab32b.slice/crio-14a8e46e320f4b6f54213a4c49c16f07c3f70dccb0aa0f529bff022144d604b4 WatchSource:0}: Error finding container 14a8e46e320f4b6f54213a4c49c16f07c3f70dccb0aa0f529bff022144d604b4: Status 404 returned error can't find the container with id 14a8e46e320f4b6f54213a4c49c16f07c3f70dccb0aa0f529bff022144d604b4 Sep 11 05:37:14.904997 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:14.904960 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerStarted","Data":"72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a"} Sep 11 05:37:14.905164 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:14.905027 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerStarted","Data":"14a8e46e320f4b6f54213a4c49c16f07c3f70dccb0aa0f529bff022144d604b4"} Sep 11 05:37:19.927027 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:19.926995 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerID="72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a" exitCode=0 Sep 11 05:37:19.927383 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:19.927069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerDied","Data":"72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a"} Sep 11 05:37:20.934323 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:20.934286 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerStarted","Data":"0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46"} Sep 11 05:37:26.964191 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:26.964105 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerID="0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46" exitCode=0 Sep 11 05:37:26.964191 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:26.964171 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerDied","Data":"0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46"} Sep 11 05:37:27.972086 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:27.972050 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerStarted","Data":"d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d"} Sep 11 05:37:27.972605 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:27.972105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerStarted","Data":"23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56"} Sep 11 05:37:27.994077 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:27.993977 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podStartSLOduration=14.993953683 podStartE2EDuration="14.993953683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:37:27.992031601 +0000 UTC m=+1021.101769768" watchObservedRunningTime="2026-09-11 05:37:27.993953683 +0000 UTC m=+1021.103691852" Sep 11 05:37:29.441948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:29.441906 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:29.442333 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:29.441985 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:29.443407 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:29.443381 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 05:37:29.443501 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:29.443385 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 11 05:37:32.442304 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:32.442256 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 11 05:37:32.442701 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:32.442266 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 05:37:34.441768 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:34.441735 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:34.442118 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:34.441791 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:35.441857 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:35.441815 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 11 05:37:35.447834 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:35.447808 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:35.451272 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:35.451251 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:38.442561 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:38.442502 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 11 05:37:41.442573 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:41.442529 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:37:41.443322 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:37:41.443301 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:39:01.543299 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:01.543266 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp"] Sep 11 05:39:01.543837 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:01.543495 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="registry" containerID="cri-o://ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6" gracePeriod=30 Sep 11 05:39:01.543837 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:01.543639 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="online" containerID="cri-o://038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364" gracePeriod=30 Sep 11 05:39:01.548922 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:01.548897 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz"] Sep 11 05:39:01.549109 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:01.549087 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="registry" containerID="cri-o://23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56" gracePeriod=30 Sep 11 05:39:01.549183 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:01.549103 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="online" containerID="cri-o://d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d" gracePeriod=30 Sep 11 05:39:02.398925 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:02.398897 2577 generic.go:356] "Generic (PLEG): container finished" podID="db485323-1f12-416e-b9d2-1225f3c1f738" containerID="038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364" exitCode=0 Sep 11 05:39:02.399031 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:02.398968 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerDied","Data":"038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364"} Sep 11 05:39:02.401177 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:02.401159 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerID="d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d" exitCode=0 Sep 11 05:39:02.401262 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:02.401194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerDied","Data":"d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d"} Sep 11 05:39:03.848681 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:03.848648 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kzchd"] Sep 11 05:39:03.849045 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:03.848838 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" podUID="d77151ac-a5a5-4502-b7f7-8c7c8f949258" containerName="postgres" containerID="cri-o://f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466" gracePeriod=30 Sep 11 05:39:03.854502 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:03.854472 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-75vjp"] Sep 11 05:39:03.854699 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:03.854674 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" podUID="5dfb58ef-9a49-4688-b5ba-393138a1ac4c" containerName="redis" containerID="cri-o://19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd" gracePeriod=30 Sep 11 05:39:04.105949 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.105926 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:39:04.108638 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.108619 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" Sep 11 05:39:04.268721 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.268690 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f72p2\" (UniqueName: \"kubernetes.io/projected/d77151ac-a5a5-4502-b7f7-8c7c8f949258-kube-api-access-f72p2\") pod \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\" (UID: \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\") " Sep 11 05:39:04.268901 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.268787 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d77151ac-a5a5-4502-b7f7-8c7c8f949258-postgresdata\") pod \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\" (UID: \"d77151ac-a5a5-4502-b7f7-8c7c8f949258\") " Sep 11 05:39:04.268901 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.268838 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2swxc\" (UniqueName: \"kubernetes.io/projected/5dfb58ef-9a49-4688-b5ba-393138a1ac4c-kube-api-access-2swxc\") pod \"5dfb58ef-9a49-4688-b5ba-393138a1ac4c\" (UID: \"5dfb58ef-9a49-4688-b5ba-393138a1ac4c\") " Sep 11 05:39:04.270974 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.270925 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfb58ef-9a49-4688-b5ba-393138a1ac4c-kube-api-access-2swxc" (OuterVolumeSpecName: "kube-api-access-2swxc") pod "5dfb58ef-9a49-4688-b5ba-393138a1ac4c" (UID: "5dfb58ef-9a49-4688-b5ba-393138a1ac4c"). InnerVolumeSpecName "kube-api-access-2swxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:39:04.270974 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.270969 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77151ac-a5a5-4502-b7f7-8c7c8f949258-kube-api-access-f72p2" (OuterVolumeSpecName: "kube-api-access-f72p2") pod "d77151ac-a5a5-4502-b7f7-8c7c8f949258" (UID: "d77151ac-a5a5-4502-b7f7-8c7c8f949258"). InnerVolumeSpecName "kube-api-access-f72p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:39:04.298095 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.298064 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77151ac-a5a5-4502-b7f7-8c7c8f949258-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "d77151ac-a5a5-4502-b7f7-8c7c8f949258" (UID: "d77151ac-a5a5-4502-b7f7-8c7c8f949258"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:39:04.374883 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.370229 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2swxc\" (UniqueName: \"kubernetes.io/projected/5dfb58ef-9a49-4688-b5ba-393138a1ac4c-kube-api-access-2swxc\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:04.374883 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.370258 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f72p2\" (UniqueName: \"kubernetes.io/projected/d77151ac-a5a5-4502-b7f7-8c7c8f949258-kube-api-access-f72p2\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:04.374883 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.370274 2577 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d77151ac-a5a5-4502-b7f7-8c7c8f949258-postgresdata\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:04.410239 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.410213 2577 generic.go:356] "Generic (PLEG): container finished" podID="d77151ac-a5a5-4502-b7f7-8c7c8f949258" containerID="f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466" exitCode=0 Sep 11 05:39:04.410358 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.410269 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" Sep 11 05:39:04.410358 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.410298 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" event={"ID":"d77151ac-a5a5-4502-b7f7-8c7c8f949258","Type":"ContainerDied","Data":"f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466"} Sep 11 05:39:04.410358 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.410341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kzchd" event={"ID":"d77151ac-a5a5-4502-b7f7-8c7c8f949258","Type":"ContainerDied","Data":"bd4e1c1d0fa379ba6c8e1d7e80d2df8c036f5722a26d7245c49139a4e228f09f"} Sep 11 05:39:04.410526 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.410364 2577 scope.go:117] "RemoveContainer" containerID="f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466" Sep 11 05:39:04.411694 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.411671 2577 generic.go:356] "Generic (PLEG): container finished" podID="5dfb58ef-9a49-4688-b5ba-393138a1ac4c" containerID="19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd" exitCode=0 Sep 11 05:39:04.411799 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.411699 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" event={"ID":"5dfb58ef-9a49-4688-b5ba-393138a1ac4c","Type":"ContainerDied","Data":"19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd"} Sep 11 05:39:04.411799 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.411725 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" Sep 11 05:39:04.411799 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.411733 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-75vjp" event={"ID":"5dfb58ef-9a49-4688-b5ba-393138a1ac4c","Type":"ContainerDied","Data":"cdb58af37a090d0922fc9c6d2c346e1afcc5d98bd172164bf8d7c6a552e92ca4"} Sep 11 05:39:04.420184 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.420167 2577 scope.go:117] "RemoveContainer" containerID="f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466" Sep 11 05:39:04.420479 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:04.420458 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466\": container with ID starting with f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466 not found: ID does not exist" containerID="f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466" Sep 11 05:39:04.420593 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.420488 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466"} err="failed to get container status \"f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466\": rpc error: code = NotFound desc = could not find container \"f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466\": container with ID starting with f3d2168432c1433ba602ed6415ffc7622687f2b34ad140e394b0e1427300a466 not found: ID does not exist" Sep 11 05:39:04.420593 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.420506 2577 scope.go:117] "RemoveContainer" containerID="19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd" Sep 11 05:39:04.427920 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.427905 2577 scope.go:117] "RemoveContainer" containerID="19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd" Sep 11 05:39:04.428134 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:04.428115 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd\": container with ID starting with 19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd not found: ID does not exist" containerID="19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd" Sep 11 05:39:04.428208 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.428141 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd"} err="failed to get container status \"19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd\": rpc error: code = NotFound desc = could not find container \"19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd\": container with ID starting with 19bd540133e8cd9cc9466693eafa8eaa1f02042ba7826e6996380db2de0ca3bd not found: ID does not exist" Sep 11 05:39:04.430314 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.430295 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kzchd"] Sep 11 05:39:04.433469 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.433449 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kzchd"] Sep 11 05:39:04.442741 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.442723 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-75vjp"] Sep 11 05:39:04.446365 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:04.446347 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-75vjp"] Sep 11 05:39:05.355328 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:05.355301 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dfb58ef-9a49-4688-b5ba-393138a1ac4c" path="/var/lib/kubelet/pods/5dfb58ef-9a49-4688-b5ba-393138a1ac4c/volumes" Sep 11 05:39:05.355729 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:05.355714 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d77151ac-a5a5-4502-b7f7-8c7c8f949258" path="/var/lib/kubelet/pods/d77151ac-a5a5-4502-b7f7-8c7c8f949258/volumes" Sep 11 05:39:05.449311 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:05.449280 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 05:39:06.595029 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:06.594984 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 05:39:15.448631 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:15.448587 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 05:39:16.594972 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:16.594933 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 05:39:25.449325 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:25.449284 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 05:39:25.449884 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:25.449425 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:39:26.594387 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:26.594346 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 05:39:26.594807 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:26.594472 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:39:32.216698 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.216658 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:39:32.219586 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.219568 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:39:32.299119 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.299091 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcrl7\" (UniqueName: \"kubernetes.io/projected/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-kube-api-access-lcrl7\") pod \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " Sep 11 05:39:32.299307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.299134 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/db485323-1f12-416e-b9d2-1225f3c1f738-feast-data\") pod \"db485323-1f12-416e-b9d2-1225f3c1f738\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " Sep 11 05:39:32.299307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.299178 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-registry-tls\") pod \"db485323-1f12-416e-b9d2-1225f3c1f738\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " Sep 11 05:39:32.299307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.299211 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-online-tls\") pod \"db485323-1f12-416e-b9d2-1225f3c1f738\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " Sep 11 05:39:32.299307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.299233 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-feast-data\") pod \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " Sep 11 05:39:32.299307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.299280 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp9sn\" (UniqueName: \"kubernetes.io/projected/db485323-1f12-416e-b9d2-1225f3c1f738-kube-api-access-pp9sn\") pod \"db485323-1f12-416e-b9d2-1225f3c1f738\" (UID: \"db485323-1f12-416e-b9d2-1225f3c1f738\") " Sep 11 05:39:32.299307 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.299305 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-registry-tls\") pod \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " Sep 11 05:39:32.299636 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.299355 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-online-tls\") pod \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\" (UID: \"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b\") " Sep 11 05:39:32.299845 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.299797 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-feast-data" (OuterVolumeSpecName: "feast-data") pod "9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" (UID: "9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:39:32.301453 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.301424 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-kube-api-access-lcrl7" (OuterVolumeSpecName: "kube-api-access-lcrl7") pod "9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" (UID: "9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b"). InnerVolumeSpecName "kube-api-access-lcrl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:39:32.301661 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.301633 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "db485323-1f12-416e-b9d2-1225f3c1f738" (UID: "db485323-1f12-416e-b9d2-1225f3c1f738"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:39:32.301760 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.301680 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-online-tls" (OuterVolumeSpecName: "online-tls") pod "db485323-1f12-416e-b9d2-1225f3c1f738" (UID: "db485323-1f12-416e-b9d2-1225f3c1f738"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:39:32.301913 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.301890 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" (UID: "9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:39:32.301996 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.301952 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-online-tls" (OuterVolumeSpecName: "online-tls") pod "9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" (UID: "9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 05:39:32.301996 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.301952 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db485323-1f12-416e-b9d2-1225f3c1f738-kube-api-access-pp9sn" (OuterVolumeSpecName: "kube-api-access-pp9sn") pod "db485323-1f12-416e-b9d2-1225f3c1f738" (UID: "db485323-1f12-416e-b9d2-1225f3c1f738"). InnerVolumeSpecName "kube-api-access-pp9sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:39:32.309012 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.308970 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db485323-1f12-416e-b9d2-1225f3c1f738-feast-data" (OuterVolumeSpecName: "feast-data") pod "db485323-1f12-416e-b9d2-1225f3c1f738" (UID: "db485323-1f12-416e-b9d2-1225f3c1f738"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:39:32.400271 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.400241 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrl7\" (UniqueName: \"kubernetes.io/projected/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-kube-api-access-lcrl7\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:32.400271 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.400268 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/db485323-1f12-416e-b9d2-1225f3c1f738-feast-data\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:32.400271 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.400277 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-registry-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:32.400497 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.400285 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/db485323-1f12-416e-b9d2-1225f3c1f738-online-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:32.400497 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.400294 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-feast-data\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:32.400497 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.400301 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp9sn\" (UniqueName: \"kubernetes.io/projected/db485323-1f12-416e-b9d2-1225f3c1f738-kube-api-access-pp9sn\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:32.400497 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.400309 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-registry-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:32.400497 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.400318 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b-online-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:39:32.536275 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.536245 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerID="23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56" exitCode=137 Sep 11 05:39:32.536450 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.536359 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" Sep 11 05:39:32.536450 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.536382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerDied","Data":"23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56"} Sep 11 05:39:32.536450 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.536441 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz" event={"ID":"9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b","Type":"ContainerDied","Data":"14a8e46e320f4b6f54213a4c49c16f07c3f70dccb0aa0f529bff022144d604b4"} Sep 11 05:39:32.536621 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.536471 2577 scope.go:117] "RemoveContainer" containerID="d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d" Sep 11 05:39:32.539062 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.539033 2577 generic.go:356] "Generic (PLEG): container finished" podID="db485323-1f12-416e-b9d2-1225f3c1f738" containerID="ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6" exitCode=137 Sep 11 05:39:32.539194 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.539105 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" Sep 11 05:39:32.539194 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.539111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerDied","Data":"ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6"} Sep 11 05:39:32.539194 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.539161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp" event={"ID":"db485323-1f12-416e-b9d2-1225f3c1f738","Type":"ContainerDied","Data":"67ece111576565b382f34164a064a1e7d9b346577f19d7afe70afbbd87b969f2"} Sep 11 05:39:32.545359 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.545344 2577 scope.go:117] "RemoveContainer" containerID="23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56" Sep 11 05:39:32.560572 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.560502 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz"] Sep 11 05:39:32.564379 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.564356 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-674b7f5594-r27wz"] Sep 11 05:39:32.565500 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.565391 2577 scope.go:117] "RemoveContainer" containerID="0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46" Sep 11 05:39:32.572749 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.572733 2577 scope.go:117] "RemoveContainer" containerID="72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a" Sep 11 05:39:32.574825 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.574804 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp"] Sep 11 05:39:32.578206 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.578181 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6f89f7d49-bxkmp"] Sep 11 05:39:32.580563 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.580532 2577 scope.go:117] "RemoveContainer" containerID="d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d" Sep 11 05:39:32.580820 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:32.580803 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d\": container with ID starting with d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d not found: ID does not exist" containerID="d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d" Sep 11 05:39:32.580869 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.580826 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d"} err="failed to get container status \"d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d\": rpc error: code = NotFound desc = could not find container \"d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d\": container with ID starting with d4c618940af23bd9bf75e32ecdee813b367c7156d6ee0053e23c545df6b8828d not found: ID does not exist" Sep 11 05:39:32.580869 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.580843 2577 scope.go:117] "RemoveContainer" containerID="23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56" Sep 11 05:39:32.581083 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:32.581065 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56\": container with ID starting with 23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56 not found: ID does not exist" containerID="23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56" Sep 11 05:39:32.581125 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.581088 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56"} err="failed to get container status \"23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56\": rpc error: code = NotFound desc = could not find container \"23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56\": container with ID starting with 23f673395820becebeb52d24d347c356306abea3bf2106d3ab41c0310d1a0f56 not found: ID does not exist" Sep 11 05:39:32.581125 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.581103 2577 scope.go:117] "RemoveContainer" containerID="0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46" Sep 11 05:39:32.581340 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:32.581325 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46\": container with ID starting with 0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46 not found: ID does not exist" containerID="0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46" Sep 11 05:39:32.581381 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.581343 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46"} err="failed to get container status \"0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46\": rpc error: code = NotFound desc = could not find container \"0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46\": container with ID starting with 0d9cd6a0e097fa558e6a7c343cda5cc6c80d6fd007636fcc4b6c3de7f8287a46 not found: ID does not exist" Sep 11 05:39:32.581381 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.581355 2577 scope.go:117] "RemoveContainer" containerID="72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a" Sep 11 05:39:32.581578 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:32.581560 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a\": container with ID starting with 72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a not found: ID does not exist" containerID="72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a" Sep 11 05:39:32.581623 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.581583 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a"} err="failed to get container status \"72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a\": rpc error: code = NotFound desc = could not find container \"72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a\": container with ID starting with 72ac95dfb0a74c129d3ce5564a11a2966634913646b6fe6fd12aa3dcf50ac16a not found: ID does not exist" Sep 11 05:39:32.581623 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.581599 2577 scope.go:117] "RemoveContainer" containerID="038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364" Sep 11 05:39:32.588778 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.588763 2577 scope.go:117] "RemoveContainer" containerID="ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6" Sep 11 05:39:32.595895 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.595879 2577 scope.go:117] "RemoveContainer" containerID="cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b" Sep 11 05:39:32.602405 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.602390 2577 scope.go:117] "RemoveContainer" containerID="596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33" Sep 11 05:39:32.608962 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.608948 2577 scope.go:117] "RemoveContainer" containerID="038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364" Sep 11 05:39:32.609227 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:32.609211 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364\": container with ID starting with 038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364 not found: ID does not exist" containerID="038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364" Sep 11 05:39:32.609276 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.609233 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364"} err="failed to get container status \"038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364\": rpc error: code = NotFound desc = could not find container \"038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364\": container with ID starting with 038fd27df2e209107edea80501d9168a739451ba751db4cf3aed4f1711793364 not found: ID does not exist" Sep 11 05:39:32.609276 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.609249 2577 scope.go:117] "RemoveContainer" containerID="ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6" Sep 11 05:39:32.609481 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:32.609464 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6\": container with ID starting with ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6 not found: ID does not exist" containerID="ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6" Sep 11 05:39:32.609563 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.609489 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6"} err="failed to get container status \"ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6\": rpc error: code = NotFound desc = could not find container \"ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6\": container with ID starting with ea8be6aff623d4d4a5b88aa54345e1131a892c931693d755254b51ddf32169d6 not found: ID does not exist" Sep 11 05:39:32.609563 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.609509 2577 scope.go:117] "RemoveContainer" containerID="cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b" Sep 11 05:39:32.609783 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:32.609767 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b\": container with ID starting with cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b not found: ID does not exist" containerID="cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b" Sep 11 05:39:32.609820 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.609788 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b"} err="failed to get container status \"cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b\": rpc error: code = NotFound desc = could not find container \"cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b\": container with ID starting with cc3fef91e88d303ec2e1a542e2e1d8c09babca768f642d98af269aa8e66c606b not found: ID does not exist" Sep 11 05:39:32.609820 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.609802 2577 scope.go:117] "RemoveContainer" containerID="596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33" Sep 11 05:39:32.610011 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:39:32.609996 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33\": container with ID starting with 596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33 not found: ID does not exist" containerID="596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33" Sep 11 05:39:32.610052 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:32.610015 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33"} err="failed to get container status \"596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33\": rpc error: code = NotFound desc = could not find container \"596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33\": container with ID starting with 596f78a10234e743d92ff0efb24b78731fbe7586eb1b7b123797d9863e8ece33 not found: ID does not exist" Sep 11 05:39:33.354859 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:33.354832 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" path="/var/lib/kubelet/pods/9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b/volumes" Sep 11 05:39:33.355322 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:39:33.355308 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" path="/var/lib/kubelet/pods/db485323-1f12-416e-b9d2-1225f3c1f738/volumes" Sep 11 05:51:12.512385 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512351 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vknlb/must-gather-k9swn"] Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512713 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfb58ef-9a49-4688-b5ba-393138a1ac4c" containerName="redis" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512724 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfb58ef-9a49-4688-b5ba-393138a1ac4c" containerName="redis" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512741 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77151ac-a5a5-4502-b7f7-8c7c8f949258" containerName="postgres" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512748 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77151ac-a5a5-4502-b7f7-8c7c8f949258" containerName="postgres" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512755 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="registry" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512761 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="registry" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512773 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="registry" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512778 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="registry" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512784 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="online" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512789 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="online" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512799 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="feast-init" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512804 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="feast-init" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512812 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="feast-apply" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512817 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="feast-apply" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512824 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="feast-init" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512829 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="feast-init" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512835 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="online" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512839 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="online" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512845 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="feast-apply" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512850 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="feast-apply" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512895 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="registry" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512904 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="registry" Sep 11 05:51:12.512898 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512910 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77151ac-a5a5-4502-b7f7-8c7c8f949258" containerName="postgres" Sep 11 05:51:12.513679 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512917 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db485323-1f12-416e-b9d2-1225f3c1f738" containerName="online" Sep 11 05:51:12.513679 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512923 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f1cd05a-b17a-4fe3-bf37-d386ea2ab32b" containerName="online" Sep 11 05:51:12.513679 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.512929 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfb58ef-9a49-4688-b5ba-393138a1ac4c" containerName="redis" Sep 11 05:51:12.515874 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.515858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:12.518470 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.518448 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vknlb\"/\"kube-root-ca.crt\"" Sep 11 05:51:12.519479 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.519457 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vknlb\"/\"default-dockercfg-p9k7v\"" Sep 11 05:51:12.519601 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.519457 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vknlb\"/\"openshift-service-ca.crt\"" Sep 11 05:51:12.524441 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.524418 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vknlb/must-gather-k9swn"] Sep 11 05:51:12.640792 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.640765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnsq\" (UniqueName: \"kubernetes.io/projected/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-kube-api-access-ljnsq\") pod \"must-gather-k9swn\" (UID: \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\") " pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:12.641009 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.640828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-must-gather-output\") pod \"must-gather-k9swn\" (UID: \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\") " pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:12.742090 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.742051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-must-gather-output\") pod \"must-gather-k9swn\" (UID: \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\") " pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:12.742232 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.742122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnsq\" (UniqueName: \"kubernetes.io/projected/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-kube-api-access-ljnsq\") pod \"must-gather-k9swn\" (UID: \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\") " pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:12.742359 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.742338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-must-gather-output\") pod \"must-gather-k9swn\" (UID: \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\") " pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:12.750117 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.750088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnsq\" (UniqueName: \"kubernetes.io/projected/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-kube-api-access-ljnsq\") pod \"must-gather-k9swn\" (UID: \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\") " pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:12.836834 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:12.836778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:13.103195 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:13.103169 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vknlb/must-gather-k9swn"] Sep 11 05:51:13.105423 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:51:13.105398 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc80b08ce_cc77_45b0_b3dd_5600ffd581ac.slice/crio-195d44d38b42555a803d77cba63824f2ec62d9de7f0885b4648667437681b7e5 WatchSource:0}: Error finding container 195d44d38b42555a803d77cba63824f2ec62d9de7f0885b4648667437681b7e5: Status 404 returned error can't find the container with id 195d44d38b42555a803d77cba63824f2ec62d9de7f0885b4648667437681b7e5 Sep 11 05:51:13.107235 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:13.107218 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 05:51:13.335786 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:13.335751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vknlb/must-gather-k9swn" event={"ID":"c80b08ce-cc77-45b0-b3dd-5600ffd581ac","Type":"ContainerStarted","Data":"195d44d38b42555a803d77cba63824f2ec62d9de7f0885b4648667437681b7e5"} Sep 11 05:51:19.368083 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:19.368043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vknlb/must-gather-k9swn" event={"ID":"c80b08ce-cc77-45b0-b3dd-5600ffd581ac","Type":"ContainerStarted","Data":"65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e"} Sep 11 05:51:20.374692 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:20.374148 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vknlb/must-gather-k9swn" event={"ID":"c80b08ce-cc77-45b0-b3dd-5600ffd581ac","Type":"ContainerStarted","Data":"5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8"} Sep 11 05:51:20.389452 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:20.389386 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vknlb/must-gather-k9swn" podStartSLOduration=2.336165945 podStartE2EDuration="8.389365875s" totalImagesPullingTime="6.05319993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 05:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:51:20.388377557 +0000 UTC m=+1853.498115724" watchObservedRunningTime="2026-09-11 05:51:20.389365875 +0000 UTC m=+1853.499104043" Sep 11 05:51:29.416053 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:29.416015 2577 generic.go:356] "Generic (PLEG): container finished" podID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" containerID="65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e" exitCode=0 Sep 11 05:51:29.416053 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:29.416048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vknlb/must-gather-k9swn" event={"ID":"c80b08ce-cc77-45b0-b3dd-5600ffd581ac","Type":"ContainerDied","Data":"65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e"} Sep 11 05:51:29.416541 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:29.416359 2577 scope.go:117] "RemoveContainer" containerID="65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e" Sep 11 05:51:29.656262 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:29.656237 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vknlb_must-gather-k9swn_c80b08ce-cc77-45b0-b3dd-5600ffd581ac/gather/0.log" Sep 11 05:51:34.959385 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:34.959353 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-vknlb/must-gather-k9swn"] Sep 11 05:51:34.960046 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:34.959567 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-vknlb/must-gather-k9swn" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" containerName="copy" containerID="cri-o://5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8" gracePeriod=2 Sep 11 05:51:34.961138 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:34.961112 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-vknlb/must-gather-k9swn"] Sep 11 05:51:34.961773 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:51:34.961737 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-k9swn\" is forbidden: User \"system:node:ip-10-0-133-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vknlb\": no relationship found between node 'ip-10-0-133-194.ec2.internal' and this object" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:35.184672 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.184651 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vknlb_must-gather-k9swn_c80b08ce-cc77-45b0-b3dd-5600ffd581ac/copy/0.log" Sep 11 05:51:35.186064 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.186046 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:35.188327 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:51:35.188302 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-k9swn\" is forbidden: User \"system:node:ip-10-0-133-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vknlb\": no relationship found between node 'ip-10-0-133-194.ec2.internal' and this object" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:35.244375 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.244322 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljnsq\" (UniqueName: \"kubernetes.io/projected/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-kube-api-access-ljnsq\") pod \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\" (UID: \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\") " Sep 11 05:51:35.244375 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.244350 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-must-gather-output\") pod \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\" (UID: \"c80b08ce-cc77-45b0-b3dd-5600ffd581ac\") " Sep 11 05:51:35.244739 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.244710 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "c80b08ce-cc77-45b0-b3dd-5600ffd581ac" (UID: "c80b08ce-cc77-45b0-b3dd-5600ffd581ac"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 05:51:35.246533 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.246504 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-kube-api-access-ljnsq" (OuterVolumeSpecName: "kube-api-access-ljnsq") pod "c80b08ce-cc77-45b0-b3dd-5600ffd581ac" (UID: "c80b08ce-cc77-45b0-b3dd-5600ffd581ac"). InnerVolumeSpecName "kube-api-access-ljnsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 05:51:35.345421 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.345385 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljnsq\" (UniqueName: \"kubernetes.io/projected/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-kube-api-access-ljnsq\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:51:35.345421 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.345416 2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c80b08ce-cc77-45b0-b3dd-5600ffd581ac-must-gather-output\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\"" Sep 11 05:51:35.354741 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.354716 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" path="/var/lib/kubelet/pods/c80b08ce-cc77-45b0-b3dd-5600ffd581ac/volumes" Sep 11 05:51:35.440272 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.440244 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vknlb_must-gather-k9swn_c80b08ce-cc77-45b0-b3dd-5600ffd581ac/copy/0.log" Sep 11 05:51:35.440609 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.440588 2577 generic.go:356] "Generic (PLEG): container finished" podID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" containerID="5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8" exitCode=143 Sep 11 05:51:35.440667 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.440647 2577 scope.go:117] "RemoveContainer" containerID="5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8" Sep 11 05:51:35.440713 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.440697 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vknlb/must-gather-k9swn" Sep 11 05:51:35.448239 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.448209 2577 scope.go:117] "RemoveContainer" containerID="65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e" Sep 11 05:51:35.458900 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.458881 2577 scope.go:117] "RemoveContainer" containerID="5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8" Sep 11 05:51:35.459199 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:51:35.459180 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8\": container with ID starting with 5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8 not found: ID does not exist" containerID="5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8" Sep 11 05:51:35.459268 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.459208 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8"} err="failed to get container status \"5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8\": rpc error: code = NotFound desc = could not find container \"5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8\": container with ID starting with 5f193a992aa0078ee5eaadae39a7b4443b3d9c4f67e2c3b1bb521154bce93dd8 not found: ID does not exist" Sep 11 05:51:35.459268 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.459230 2577 scope.go:117] "RemoveContainer" containerID="65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e" Sep 11 05:51:35.459506 ip-10-0-133-194 kubenswrapper[2577]: E0911 05:51:35.459488 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e\": container with ID starting with 65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e not found: ID does not exist" containerID="65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e" Sep 11 05:51:35.459634 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:35.459516 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e"} err="failed to get container status \"65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e\": rpc error: code = NotFound desc = could not find container \"65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e\": container with ID starting with 65a9dfdaf012140f9063ac44b2ac4a1124fca1749902a41745144cb87487c30e not found: ID does not exist" Sep 11 05:51:36.215907 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:36.215861 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5fkvd_a540b042-b24e-416e-b30a-84346d4dd47f/global-pull-secret-syncer/0.log" Sep 11 05:51:36.384254 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:36.384226 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gvw7p_4ad2bdd1-8798-4983-9878-0060b20d0e3e/konnectivity-agent/0.log" Sep 11 05:51:36.433421 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:36.433393 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-194.ec2.internal_bee55f8a1340335ec8a8808d97d355d4/haproxy/0.log" Sep 11 05:51:39.463760 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.463733 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f71e5ddf-1f2c-49a1-be8d-7428d1cec010/alertmanager/0.log" Sep 11 05:51:39.486872 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.486850 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f71e5ddf-1f2c-49a1-be8d-7428d1cec010/config-reloader/0.log" Sep 11 05:51:39.514339 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.514316 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f71e5ddf-1f2c-49a1-be8d-7428d1cec010/kube-rbac-proxy-web/0.log" Sep 11 05:51:39.536511 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.536493 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f71e5ddf-1f2c-49a1-be8d-7428d1cec010/kube-rbac-proxy/0.log" Sep 11 05:51:39.562696 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.562677 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f71e5ddf-1f2c-49a1-be8d-7428d1cec010/kube-rbac-proxy-metric/0.log" Sep 11 05:51:39.590272 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.590248 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f71e5ddf-1f2c-49a1-be8d-7428d1cec010/prom-label-proxy/0.log" Sep 11 05:51:39.615080 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.615063 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f71e5ddf-1f2c-49a1-be8d-7428d1cec010/init-config-reloader/0.log" Sep 11 05:51:39.699277 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.699257 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bgz4c_c896934d-7953-4886-b378-0db8bf0f757a/kube-state-metrics/0.log" Sep 11 05:51:39.727005 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.726939 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bgz4c_c896934d-7953-4886-b378-0db8bf0f757a/kube-rbac-proxy-main/0.log" Sep 11 05:51:39.756980 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.756962 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bgz4c_c896934d-7953-4886-b378-0db8bf0f757a/kube-rbac-proxy-self/0.log" Sep 11 05:51:39.812531 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.812513 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d57747d66-gjmx4_b344eedc-9880-44ab-8228-5df3091d8924/metrics-server/0.log" Sep 11 05:51:39.844833 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:39.844808 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-w7flr_38ca1c4e-9a5b-4447-8fa3-eb0d18f7c5ad/monitoring-plugin/0.log" Sep 11 05:51:40.062195 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.062175 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m79z4_d57b147a-1998-4dd1-af68-f081b962d8af/node-exporter/0.log" Sep 11 05:51:40.087314 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.087293 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m79z4_d57b147a-1998-4dd1-af68-f081b962d8af/kube-rbac-proxy/0.log" Sep 11 05:51:40.111270 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.111247 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m79z4_d57b147a-1998-4dd1-af68-f081b962d8af/init-textfile/0.log" Sep 11 05:51:40.142408 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.142388 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-9j4z8_7d9d622e-ff22-47a3-93e7-a75acf4c2500/kube-rbac-proxy-main/0.log" Sep 11 05:51:40.163938 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.163917 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-9j4z8_7d9d622e-ff22-47a3-93e7-a75acf4c2500/kube-rbac-proxy-self/0.log" Sep 11 05:51:40.189465 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.189445 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-9j4z8_7d9d622e-ff22-47a3-93e7-a75acf4c2500/openshift-state-metrics/0.log" Sep 11 05:51:40.252772 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.252753 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b9076e4-d0e2-4a09-a7d0-0012788e8eb4/prometheus/0.log" Sep 11 05:51:40.276420 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.276363 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b9076e4-d0e2-4a09-a7d0-0012788e8eb4/config-reloader/0.log" Sep 11 05:51:40.299123 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.299105 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b9076e4-d0e2-4a09-a7d0-0012788e8eb4/thanos-sidecar/0.log" Sep 11 05:51:40.319581 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.319542 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b9076e4-d0e2-4a09-a7d0-0012788e8eb4/kube-rbac-proxy-web/0.log" Sep 11 05:51:40.342023 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.342002 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b9076e4-d0e2-4a09-a7d0-0012788e8eb4/kube-rbac-proxy/0.log" Sep 11 05:51:40.364882 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.364860 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b9076e4-d0e2-4a09-a7d0-0012788e8eb4/kube-rbac-proxy-thanos/0.log" Sep 11 05:51:40.385906 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.385884 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b9076e4-d0e2-4a09-a7d0-0012788e8eb4/init-config-reloader/0.log" Sep 11 05:51:40.417870 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.417850 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-44bvn_a2116f63-04d2-4c2c-9882-878e5079dc7c/prometheus-operator/0.log" Sep 11 05:51:40.440764 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.440730 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-44bvn_a2116f63-04d2-4c2c-9882-878e5079dc7c/kube-rbac-proxy/0.log" Sep 11 05:51:40.468375 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.468360 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-jpwsx_64e7d3d0-8e1e-4ec6-9755-d8313f2da409/prometheus-operator-admission-webhook/0.log" Sep 11 05:51:40.502904 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.502888 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-687649496d-nfrc9_e11a7a4e-5c36-46e7-890b-be844875efb3/telemeter-client/0.log" Sep 11 05:51:40.522844 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.522826 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-687649496d-nfrc9_e11a7a4e-5c36-46e7-890b-be844875efb3/reload/0.log" Sep 11 05:51:40.543005 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.542952 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-687649496d-nfrc9_e11a7a4e-5c36-46e7-890b-be844875efb3/kube-rbac-proxy/0.log" Sep 11 05:51:40.574072 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.574051 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4d5ff898-l2gp9_48572d15-cecc-4119-8ae4-519cbd0b0b0a/thanos-query/0.log" Sep 11 05:51:40.595172 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.595155 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4d5ff898-l2gp9_48572d15-cecc-4119-8ae4-519cbd0b0b0a/kube-rbac-proxy-web/0.log" Sep 11 05:51:40.615323 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.615303 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4d5ff898-l2gp9_48572d15-cecc-4119-8ae4-519cbd0b0b0a/kube-rbac-proxy/0.log" Sep 11 05:51:40.639974 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.639958 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4d5ff898-l2gp9_48572d15-cecc-4119-8ae4-519cbd0b0b0a/prom-label-proxy/0.log" Sep 11 05:51:40.664977 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.664961 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4d5ff898-l2gp9_48572d15-cecc-4119-8ae4-519cbd0b0b0a/kube-rbac-proxy-rules/0.log" Sep 11 05:51:40.692008 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:40.691987 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4d5ff898-l2gp9_48572d15-cecc-4119-8ae4-519cbd0b0b0a/kube-rbac-proxy-metrics/0.log" Sep 11 05:51:43.657572 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.657517 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn"] Sep 11 05:51:43.658073 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.658021 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" containerName="gather" Sep 11 05:51:43.658073 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.658039 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" containerName="gather" Sep 11 05:51:43.658073 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.658062 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" containerName="copy" Sep 11 05:51:43.658073 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.658071 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" containerName="copy" Sep 11 05:51:43.658341 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.658145 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" containerName="gather" Sep 11 05:51:43.658341 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.658164 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80b08ce-cc77-45b0-b3dd-5600ffd581ac" containerName="copy" Sep 11 05:51:43.661500 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.661478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.663983 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.663963 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbwzt\"/\"kube-root-ca.crt\"" Sep 11 05:51:43.664089 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.664000 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbwzt\"/\"openshift-service-ca.crt\"" Sep 11 05:51:43.665072 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.665056 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gbwzt\"/\"default-dockercfg-m9v22\"" Sep 11 05:51:43.668620 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.668333 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn"] Sep 11 05:51:43.710983 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.710961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-podres\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.711086 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.710991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwg9m\" (UniqueName: \"kubernetes.io/projected/5c589540-080c-453b-bf04-ade3de0ba418-kube-api-access-fwg9m\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.711086 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.711025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-proc\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.711086 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.711044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-sys\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.711200 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.711097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-lib-modules\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.812445 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.812409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-podres\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.812445 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.812452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwg9m\" (UniqueName: \"kubernetes.io/projected/5c589540-080c-453b-bf04-ade3de0ba418-kube-api-access-fwg9m\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.812628 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.812485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-proc\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.812628 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.812503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-sys\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.812628 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.812535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-lib-modules\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.812628 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.812610 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-podres\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.812784 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.812625 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-proc\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.812784 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.812632 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-sys\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.812784 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.812679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c589540-080c-453b-bf04-ade3de0ba418-lib-modules\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.820224 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.820205 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwg9m\" (UniqueName: \"kubernetes.io/projected/5c589540-080c-453b-bf04-ade3de0ba418-kube-api-access-fwg9m\") pod \"perf-node-gather-daemonset-tl5qn\" (UID: \"5c589540-080c-453b-bf04-ade3de0ba418\") " pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:43.972273 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:43.972218 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:44.239039 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:44.239016 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn"] Sep 11 05:51:44.241364 ip-10-0-133-194 kubenswrapper[2577]: W0911 05:51:44.241326 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c589540_080c_453b_bf04_ade3de0ba418.slice/crio-0e6304e1b232270a7e7a7256543f46231838b5394117b2623f0d13bcdb4b63d4 WatchSource:0}: Error finding container 0e6304e1b232270a7e7a7256543f46231838b5394117b2623f0d13bcdb4b63d4: Status 404 returned error can't find the container with id 0e6304e1b232270a7e7a7256543f46231838b5394117b2623f0d13bcdb4b63d4 Sep 11 05:51:44.315948 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:44.315930 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kw5tg_f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0/dns/0.log" Sep 11 05:51:44.360892 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:44.360865 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kw5tg_f7b7e3b3-9dbb-4f5a-8476-60ee7e55aee0/kube-rbac-proxy/0.log" Sep 11 05:51:44.473348 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:44.473319 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lk5n4_3fa8974b-7bf1-4d99-884d-d5e923f8e568/dns-node-resolver/0.log" Sep 11 05:51:44.477407 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:44.477381 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" event={"ID":"5c589540-080c-453b-bf04-ade3de0ba418","Type":"ContainerStarted","Data":"a4ee96390da4344d3af8cfab62d7b78ffb38a97fa1b43e5488702787f7581bb5"} Sep 11 05:51:44.477530 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:44.477425 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" event={"ID":"5c589540-080c-453b-bf04-ade3de0ba418","Type":"ContainerStarted","Data":"0e6304e1b232270a7e7a7256543f46231838b5394117b2623f0d13bcdb4b63d4"} Sep 11 05:51:44.477530 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:44.477488 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:44.492817 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:44.492692 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" podStartSLOduration=1.492676072 podStartE2EDuration="1.492676072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 05:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 05:51:44.49240219 +0000 UTC m=+1877.602140357" watchObservedRunningTime="2026-09-11 05:51:44.492676072 +0000 UTC m=+1877.602414239" Sep 11 05:51:44.905779 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:44.905754 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvntc_d4d6f060-e767-49e2-8c41-a32d0b1a83ae/node-ca/0.log" Sep 11 05:51:45.915396 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:45.915368 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jbzzm_7232efa1-0a8b-47fe-a41a-55f84611ff7e/serve-healthcheck-canary/0.log" Sep 11 05:51:46.423066 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:46.423033 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vch8h_4dab642a-b903-4531-a22a-c796d043aaac/kube-rbac-proxy/0.log" Sep 11 05:51:46.441969 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:46.441944 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vch8h_4dab642a-b903-4531-a22a-c796d043aaac/exporter/0.log" Sep 11 05:51:46.461109 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:46.461087 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vch8h_4dab642a-b903-4531-a22a-c796d043aaac/extractor/0.log" Sep 11 05:51:50.491612 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:50.491534 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gbwzt/perf-node-gather-daemonset-tl5qn" Sep 11 05:51:51.469078 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:51.469054 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6th6w_830c602d-f84b-4211-8203-78d39e810f0a/kube-multus/0.log" Sep 11 05:51:51.492040 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:51.492022 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2svzc_2010dca1-86fb-471b-8a76-b9ff03a30d37/kube-multus-additional-cni-plugins/0.log" Sep 11 05:51:51.513490 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:51.513473 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2svzc_2010dca1-86fb-471b-8a76-b9ff03a30d37/egress-router-binary-copy/0.log" Sep 11 05:51:51.538313 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:51.538297 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2svzc_2010dca1-86fb-471b-8a76-b9ff03a30d37/cni-plugins/0.log" Sep 11 05:51:51.557894 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:51.557873 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2svzc_2010dca1-86fb-471b-8a76-b9ff03a30d37/bond-cni-plugin/0.log" Sep 11 05:51:51.579790 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:51.579777 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2svzc_2010dca1-86fb-471b-8a76-b9ff03a30d37/routeoverride-cni/0.log" Sep 11 05:51:51.599413 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:51.599398 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2svzc_2010dca1-86fb-471b-8a76-b9ff03a30d37/whereabouts-cni-bincopy/0.log" Sep 11 05:51:51.620361 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:51.620347 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2svzc_2010dca1-86fb-471b-8a76-b9ff03a30d37/whereabouts-cni/0.log" Sep 11 05:51:52.041233 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:52.041210 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hqmnw_ac746c00-7462-4686-9f96-7d5be8804348/network-metrics-daemon/0.log" Sep 11 05:51:52.063795 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:52.063770 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hqmnw_ac746c00-7462-4686-9f96-7d5be8804348/kube-rbac-proxy/0.log" Sep 11 05:51:53.335169 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:53.335137 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-srxzk_f8b8a658-19bf-4a12-9a6a-e348e106b6d3/ovn-controller/0.log" Sep 11 05:51:53.352957 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:53.352931 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-srxzk_f8b8a658-19bf-4a12-9a6a-e348e106b6d3/ovn-acl-logging/0.log" Sep 11 05:51:53.374327 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:53.374301 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-srxzk_f8b8a658-19bf-4a12-9a6a-e348e106b6d3/kube-rbac-proxy-node/0.log" Sep 11 05:51:53.393297 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:53.393278 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-srxzk_f8b8a658-19bf-4a12-9a6a-e348e106b6d3/kube-rbac-proxy-ovn-metrics/0.log" Sep 11 05:51:53.412990 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:53.412933 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-srxzk_f8b8a658-19bf-4a12-9a6a-e348e106b6d3/northd/0.log" Sep 11 05:51:53.433666 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:53.433648 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-srxzk_f8b8a658-19bf-4a12-9a6a-e348e106b6d3/nbdb/0.log" Sep 11 05:51:53.454007 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:53.453993 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-srxzk_f8b8a658-19bf-4a12-9a6a-e348e106b6d3/sbdb/0.log" Sep 11 05:51:53.494215 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:53.494197 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-srxzk_f8b8a658-19bf-4a12-9a6a-e348e106b6d3/ovnkube-controller/0.log" Sep 11 05:51:54.454790 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:54.454760 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-g8zdr_e699639b-b57e-4d9a-93b9-ec9b71a507fc/network-check-target-container/0.log" Sep 11 05:51:55.357372 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:55.357349 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-m7b82_d1f7ce08-a06a-4063-92e7-8f2c02044ba8/iptables-alerter/0.log" Sep 11 05:51:56.010589 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:56.010546 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-fzhcj_05a69a1a-eb1d-4eaa-9ab0-138e8b11bb95/tuned/0.log" Sep 11 05:51:59.093912 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:59.093881 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-8bp9k_130e718b-93e4-4541-9a2a-9ee22ac205de/csi-driver/0.log" Sep 11 05:51:59.115860 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:59.115834 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-8bp9k_130e718b-93e4-4541-9a2a-9ee22ac205de/csi-node-driver-registrar/0.log" Sep 11 05:51:59.138295 ip-10-0-133-194 kubenswrapper[2577]: I0911 05:51:59.138268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-8bp9k_130e718b-93e4-4541-9a2a-9ee22ac205de/csi-liveness-probe/0.log"